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NATO  Independent  Cost  Estimating  and 
the  Role  of  Life  Cycle  Cost  Analysis  in 
Managing  the  Defence  Enterprise 
(RTO-TR-SAS-076) 

Executive  Summary 

At  its  October  2006  Panel  Business  Meeting  (PBM),  the  Systems  Analysis  and  Studies  (SAS)  Panel 
established  an  Exploratory  Team  (ET.AY)  to  propose  new  life-cycle  cost  estimating  activities  as  follow-on 
to  the  SAS-054  Task  Group  on  Methods  and  Models  for  Life-Cycle  Costing.  SAS-054,  in  turn,  built  on  the 
work  of  SAS-028,  which  had  developed  a  generic  cost  breakdown  structure.  In  addition  to  this  work  a  code 
practice  was  developed  by  SAS-069.  One  of  the  activities  the  Exploratory  Team  recommended,  as  the  next 
logical  phase  that  would  build  upon  the  efforts  of  SAS-028  and  SAS-054,  is  to  prove  the  concepts  of  these 
earlier  RTO  SAS  works  on  Life  Cycle  Cost  Analysis  by  rigorously  applying  the  guidelines  to  generate 
sound,  reliable,  Independent  Cost  Estimates  (ICE)  of  a  major  weapon  system  acquisition  programs  of 
current,  international  interest: 

•  HMS  Rotterdam  and  HMS  Johan  de  Witt  Landing  Platform  Docks  (LPD)  of  the  Netherlands;  and 

•  NATO’s  Alliance  Ground  Surveillance  System  (AGS). 

Life-cycle  cost  estimates  of  defence  acquisition  programs  are  inherently  uncertain.  Estimates  are  sometimes 
required  when  little  if  any  of  a  program’s  total  cost  is  known.  Years  of  system  development  and  production, 
and  decades  of  operating  and  support  costs,  need  to  be  estimated.  Estimates,  in  turn,  are  based  on  historical 
samples  of  data  that  are  almost  always  messy,  of  limited  size,  and  difficult  and  costly  to  obtain.  Further,  key 
characteristics  of  the  system  may  actually  change  as  the  system  proceeds  through  development  and  even 
production.  Increases  in  system  weight,  complexity,  and  lines  of  code  are  commonplace.  Innovative, 
pioneering  statistical  techniques  were  employed  in  these  efforts  that  pushed  the  state  of  the  art  in  cost 
analysis  to  new  frontiers: 

•  Decision  trees,  regression  models,  and  hierarchical  clustering,  based  on  scores  of  technical  and 
performance  characteristics,  to  estimate  ship  acquisition  costs.  The  task  group  produced  the  paper 
“An  Application  of  Data  Mining  Algorithms  for  Shipbuilding  Cost  Estimation”  which  won  “Best 
Paper”  in  the  Applications  Track  at  a  major  international  symposium  and  was  published  in 
professional  literature. 

•  Benchmark  coefficients  of  variation  coupled  with  a  point  estimate  to  generate  a  cumulative 
probability  distribution  of  cost,  or  S-curve,  for  NATO  AGS.  This  work  resonated  strongly  throughout 
the  defence  cost  community  and  directly  supported  the  Alliance’s  acquisition  of  this  critical  asset. 

SAS-076  also  ventured  into  the  realm  of  defence  planning  and  acquisition.  Every  NATO  and  Partnership 
for  Peace  (PfP)  Nation  has  its  own  procedures  and  processes  for  establishing  national  security  guidance 
and  objectives;  for  identifying  military  needs;  and  for  developing  and  procuring  solutions.  Within  the 
context  of  this  melange  of  national  norms,  SAS-076: 

•  Conducted  an  international  conference  on  capability  portfolio  analysis; 

•  Captured  commonalities  and  differences  in  national  processes  and  procedures;  and 

•  Recommended  a  set  of  best  practices  in  the  cost-analysis  domain. 


RTO-TR-SAS-076 


ES  - 1 


By  working  as  a  cohesive  team  under  the  aegis  of  the  Systems  Analysis  and  Studies  Panel,  SAS-076 
demonstrated  that  strong  results  can  be  achieved  and  significant  value  added  for  the  entire  NATO  and  PfP 
defense  community. 
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Estimation  independante  des  couts  de  l’OTAN 
et  role  de  l’analyse  des  couts  globaux  de 
possession  au  sein  de  l’OTAN 
(RTO-TR-SAS-076) 


Synthese 

Lors  de  sa  reunion  d’octobre  2006,  la  Commission,  sur  les  Etudes  et  1’ Analyse  de  Systeme  (SAS)  a  cree 
l’Equipe  exploratoire  (EE.AY)  en  vue  de  proposer  de  nouvelles  activites  pour  l’estimation  des  couts  globaux 
de  possession  dans  le  prolongement  du  groupe  de  travail  SAS-028  sur  les  methodes  et  les  modeles 
devaluation  des  couts  globaux  de  possession.  A  son  tour,  SAS-054  s’est  appuye  sur  les  travaux  de  SAS-028, 
qui  avait  developpe  une  structure  generique  de  ventilation  des  couts.  En  complement  de  ce  travail,  SAS-069 
avait  developpe  un  code  de  bonnes  pratiques.  Une  des  activites  recommandees  par  Tequipe  exploratoire, 
consideree  comme  phase  suivante  logique  s’appuyant  sur  les  efforts  realises  par  SAS-028  et  SAS-054, 
consiste  a  demontrer  la  validite  des  concepts  de  ces  travaux  preliminaires  relatifs  a  E  analyse  des  couts 
globaux  de  possession  en  se  basant  sur  une  application  rigoureuse  des  directives,  en  vue  de  produire  des 
Estimations  de  Couts  Independantes  (ECI)  justes  et  fiables  concemant  des  programmes  d’acquisition  d’un 
systeme  d’anne  principal  d’envergure  intemationale  et  actuelle  : 

•  Les  plateformes  d’atterrissage  du  EIMS  Rotterdam  et  du  EIMS  Johan  de  Witt  des  Pays-Bas  ;  et 

•  Le  systeme  de  surveillance  terrestre  Alliance  Ground  Sun’eillance  (AGS)  de  l’OTAN. 

Les  estimations  des  couts  globaux  de  possession  des  programmes  d’acquisition  de  defense  sont  par  nature 
incertaines.  Ces  estimations  sont  parfois  necessaires  lorsque  le  cout  total  d’un  programme  n’est  connu  qu’en 
infime  partie  ou  pas  du  tout.  Les  annees  dediees  au  developpement  du  systeme  et  a  la  production,  ainsi  que 
les  couts  d’exploitation  et  de  soutien  sur  des  dizaines  d’annees  doivent  etre  evalues.  Les  evaluations,  a  leur 
tour,  sont  basees  sur  un  echantillonnage  de  donnees  historiques  qui  sont  la  plupart  du  temps  desordonnees  et 
de  faible  volume,  et  par  ailleurs  difficiles  et  couteuses  a  obtenir.  En  outre,  les  caracteristiques  principals  du 
systeme  peuvent  en  fait  evoluer  en  fonction  de  la  progression  du  systeme  dans  la  phase  de  developpement  et 
parfois  meme  de  production.  L’ augmentation  du  poids,  de  la  complexity  et  des  lignes  de  code  du  systeme  est 
courante.  Des  techniques  statistiques  innovantes  et  pionnieres  ont  ete  integrees  dans  ces  travaux  qui  ont  fait 
reculer  les  limites  d’une  analyse  des  couts  a  la  pointe  du  progres  : 

•  Les  arbres  decisionnels,  les  modeles  de  regression,  et  le  regroupement  hierarchique,  bases  sur  de 
nombreuses  caracteristiques  techniques  et  de  performance,  pour  realiser  l’estimation  des  couts 
d’acquisition  d’un  navire.  Le  groupe  de  travail  a  redige  un  document  intitule  «  Application  des 
algorithmes  d’ extraction  de  donnees  pour  l’estimation  des  couts  dans  la  construction  navale  »  qui 
a  remporte  le  prix  du  «  Meilleur  Document  »  dans  les  Orientations  d’Applications  lors  d’un  grand 
symposium  international,  et  qui  a  ete  publie  dans  des  ouvrages  specialisees. 

•  Les  coefficients  de  reference  de  la  variation  associee  a  une  estimation  ponctuelle  en  vue  de  generer 
une  distribution  stochastique  cumulative  du  cout,  ou  courbe  en  S,  pour  LAGS  de  l’OTAN.  Ces  travaux 
ont  eu  un  impact  important  a  travers  toute  la  communaute  chargee  des  couts  de  defense  et  ont 
apporte  un  soutien  direct  a  1’ Alliance  dans  sa  demarche  d’acquisition  de  ce  systeme  critique. 

SAS-076  s’est  egalement  aventure  dans  le  vaste  domaine  de  la  planification  et  l’acquisition  de  defense. 
Chaque  membre  de  l’OTAN  et  du  Partenariat  pour  la  Paix  (PpP)  possede  ses  propres  procedures  et 
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processus  pour  l’etablissement  des  orientations  et  des  objectifs  en  matiere  de  securite  nationale  ; 
1’ identification  des  besoins  militaires  ;  et  le  developpement  et  f  appro visionnement  en  solutions.  Dans  ce 
contexte  de  diversite  entermes  de  normes  nationales,  SAS-076  : 

•  A  organise  une  conference  intemationale  sur  l’analyse  de  l’eventail  des  capacites  ; 

•  A  collecte  les  similitudes  et  les  differences  entre  les  procedures  et  les  processus  nationaux  ;  et 

•  A  recommande  un  ensemble  de  bonnes  pratiques  a  respecter  dans  le  domaine  de  f  analyse  des 
couts. 

Sous  la  supervision  de  la  Commission  sur  les  Etudes  et  l’Analyse  de  Systeme  l’equipe  SAS-076  a  realise 
un  travail  d’ensemble  et  a  demontre  qu’il  etait  possible  d’obtenir  des  resultats  de  qualite  et  d’apporter  une 
valeur  ajoutee  significative  a  l’ensemble  de  la  communaute  de  defense  de  l’OTAN  et  des  etats  du  PpP. 
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NATO  INDEPENDENT  COST  ESTIMATING  AND  THE  ROLE  OF  LIFE 
CYCLE  COST  ANALYSIS  IN  MANAGING  THE  DEFENCE  ENTERPRISE 


1.0  INTRODUCTION 

1.1  Background 

Life-cycle  cost  estimates  of  defense  acquisition  programs  are  inherently  uncertain.  Estimates  are  often  required 
when  only  5%  of  a  program’s  total  cost  is  known.  Years  of  system  development  and  production,  and  decades 
of  operating  and  support  costs,  need  to  be  estimated.  Estimates,  in  turn,  are  based  on  historical  samples  of  data 
that  are  almost  always  messy,  of  limited  size,  and  difficult  and  costly  to  obtain.  Herculean  efforts  are  commonly 
required  to  squeeze  usable  information  from  a  limited,  inconsistent  set  of  data.  And  no  matter  what  estimation 
tool  or  method  is  used,  historical  observations  never  perfectly  fit  a  smooth  line  or  surface  but  instead  fall  above 
and  below  an  estimated  value.  To  complicate  matters,  the  weapon  system  or  automated  information  system 
under  study  is  often  of  sketchy  design.  Only  limited  programmatic  information  may  be  available  on  such  key 
parameters  as  schedule,  quantity  of  units  to  be  bought,  performance,  requirements,  acquisition  strategy,  and 
future  evolutionary  increments.  Further,  key  characteristics  of  the  system  may  actually  change  as  the  system 
proceeds  through  development  and  even  production.  Increases  in  system  weight,  complexity,  and  lines  of  code 
are  commonplace. 

For  all  of  these  reasons,  a  life-cycle  cost  estimate,  when  expressed  as  a  single  number,  is  merely  one 
outcome  or  observation  in  a  probability  distribution  of  costs.  That  is,  the  estimate  is  stochastic  rather  than 
deterministic,  with  uncertainty  and  risk  determining  the  shape  and  variance  of  the  distribution. 

Given  the  difficult  job  of  producing  credible,  cradle-to-grave  cost  estimates  in  a  highly  stochastic  acquisition 
and  estimating  environment,  SAS-054  has  developed  a  guideline  to  conduct  a  life-cycle  cost  analysis  in  a 
multi-national  environment.  SAS-076  then  sought  to  establish  a  new  mode  of  cooperative  working  within  the 
international  cost-estimating  community.  Hitherto,  nations  have  often  developed,  by  themselves,  life-cycle  cost 
estimates  for  acquisition  programs  of  mutual  or  collaborative  interest. 

The  vision  of  SAS-076  is  that  by  working  together  in  the  cost-estimation  process,  NATO  and  PfP  nations 
will  all  hopefully  benefit  from  a  cross-fertilization  of  ideas,  data,  experiences,  perspectives  in  multi-national 
projects. 

1.2  Provenance  and  Approval 

The  NATO  Research  &  Technology  Organization  (RTO)  System  Analysis  and  Studies  (SAS)  Task  Group  SAS- 
054  (Methods  and  Models  for  Life  Cycle  Costing)  published  guidelines  for  conducting  life  cycle  costing  analy¬ 
sis  within  the  NATO  nations  [1],  Their  comprehensive  report  provided  an  extensive  review  of  cost  forecasting 
methods  and  models,  including  the  two  most  critical  elements  for  any  viable  forecasting  analysis:  data  collec¬ 
tion  and  the  measurement  of  uncertainty  and  risk.  One  of  the  key  recommendations  of  the  study  would  be  to 
demonstrate  the  proof  of  concept  (methods  and  models)  described  in  the  report  by  using  a  practical  application 
of  the  guideline”. 

At  its  October  2006  Panel  Business  Meeting  (PBM),  the  Systems  Analysis  and  Studies  (SAS)  Panel  es¬ 
tablished  an  Exploratory  Team  (ET.AY)  to  propose  new  life  cycle  cost  estimating  activities  as  a  follow-on  to 
SAS-0541.  The  Exploratory  Team  was  established  to  determine  the  feasibility  of  conducting  an  independent 
cost  estimate  (ICE)  of  a  major  weapon  system  acquisition  programme  of  current  international  interest.  Candi¬ 
date  systems  discussed  by  the  team  members  included  NATO  AGS,  NATO’s  multi-mission  aircraft,  A400M,  the 
European  multi-mission  frigate,  FREMM,  amongst  others.  Within  a  year  of  establishment,  the  team  found  just 
how  hard  it  was  to  get  approval  from  national  consortia  to  develop  an  ICE  for  a  major  acquisition  programme. 

1  SAS-054,  in  turn,  built  on  the  work  of  SAS-028  which  had  developed  a  generic  cost  breakdown  structure. 
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Rejection  notes  were  received  confirming  that  no  approval  could  be  given  involving  a  study  on  FREMM  and 
A400M.  Flowever,  approval  was  received  for  the  team  to  conduct  an  ICE  on  the  NATO  AGS  programme,  and 
eventually,  for  the  Royal  Netherlands  Navy  Landing  Platform  Dock  Rotterdam  class  amphibious  warfare  vessel 
and  the  Canadian  LAV  III,  light  armoured  vehicle. 

The  Exploratory  Team  recommended  a  third  and  logical  phase  that  would  build  upon  the  efforts  of  SAS-028 
and  SAS-054: 

•  Implement  best  life  cycle  costing  practices,  as  defined  in  the  work  of  SAS-054,  by  rigorously  applying 
them  to  generate  a  sound,  reliable  independent  cost  estimate  (ICE)  of  a  major  weapon  system  acquisition 
programme  of  current,  international  interest.  This  will  demonstrate  proof  of  concept  of  the  guidelines. 

•  Identify  best  practices  in  capability  portfolio  analysis,  with  an  emphasis  on  the  role  of  cost  analysis  in 
the  process. 

NATO’s  Research  and  Technology  Board  (RTB)  approved  these  recommendations  in  the  Spring  of  2008 
and  stood-up  SAS-076  to  implement  them.  In  June  2008,  the  first  meeting  of  SAS  076,  NATO  Independent 
Cost  Estimating  and  its  Role  in  Capability  Portfolio  Analysis,  was  held  in  Rome,  Italy. 

1.3  Objectives 

Study  objectives  were  defined  as  follows: 

•  Independent  Cost  Estimates  (ICEs) 

-  Choose  one  or  more  existing  systems  for  which  there  is  a  CARD  (Cost  Analysis  Requirements 
Description  -  used  in  the  U.S.)  or  an  MDAL  (Master  Data  and  Assumptions  List  -  used  in  the  U.K.) 
and  for  which  actual  cost  data  exist  for  development,  production  and  in-service. 

-  Generate  an  independent  cost  estimate  based  on  the  SAS-054  guidelines  for  best  practice,  using  the 
CARD  or  MDAL  as  it  existed  at  the  start  of  the  acquisition  program. 

-  Analyze  risks  and  uncertainties. 

-  Generate  costs  over  the  life  cycle. 

-  Obtain  information  on  the  actual  cost  of  the  weapon  system  under  study,  if  available.  Tally  and  ana¬ 
lyze  differences  between  actuals  and  estimates,  thus  providing  invaluable  feedback  on  the  accuracy 
and  completeness  of  the  guidelines. 

-  Provide  a  complete,  documented  analysis  for  both  ex-post  and  ex-ante  cost  estimation  to  decision 
makers. 

Through  these  activities,  the  cost  estimating  concepts  of  the  guideline  developed  in  the  SAS-054  study 
activity  were  to  be  demonstrated  and  validated,  including  the  methods,  models  and  processes. 

•  Role  of  Cost  Analysis  in  Managing  the  Defense  Enterprise 

Most  defense  establishments  are  quite  adept  at  measuring  the  cost  and  the  value  of  a  specific  program  to 
fulfill  a  specified  mission.  Trade-offs  arc  conducted.  Analyses  of  alternatives  are  studied.  Sometimes, 
gap  analysis  is  done.  But  are  comparisons  made  program  versus  program  with  an  emphasis  on  the  best 
way  of  meeting  national  and  coalition  objectives?  Portfolio  analysis  is  a  promising  method  to  improve 
defense  business  practices  by  analyzing  a  group  of  systems  as  a  whole  rather  than  focusing  on  acquisition 
programs  one  at  a  time. 

Specific  objectives  of  SAS-076  were  to: 
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Figure  1 :  The  life  cycle  cost  iceberg 


-  Identify  commonalities  and  differences  among  NATO  and  PfP  nations  in  executing  capability  port¬ 
folio  analysis,  especially  with  respect  to  the  use  of  life-cycle  cost  analysis. 

-  Recommend  best  practices  regarding  the  analysis  of  risks  and  uncertainties  of  costs,  capabilities, 
and  requirements,  thus  engendering  more  informed  decision-making  on  the  allocation  of  scarce 
defense  resources. 

1.4  Scope 

ICEs  were  generated  for  these  acquisition  programs: 

•  NATO  Alliance  Ground  Surveillance  System  (AGS) 

•  Royal  Netherlands  Navy  Landing  Platform  Dock  (LPD):  HMS  Rotterdam  and  HMS  Johan  de  Witt 

Within  the  domain  of  total  ownership  costs  shown  in  Figure  1 ,  the  primary  focus  of  the  ICEs  was  estimation 
of  acquisition  costs,  and,  in  some  cases,  operating  and  support  costs.  Infrastructure  costs  were  beyond  the  scope 
of  this  effort.  Details  on  the  AGS  ICE  are  presented  in  Annex  A.  Details  on  the  Netherlands  LPD  ICEs  are 
presented  in  Annex  B1  and  B2. 

The  effort  in  capability  portfolio  analysis  was  expanded  to  include  an  investigation  of  the  role  played  by 
life-cycle  cost  analysis  in  managing  the  entire  defense  enterprise,  as  shown  in  Figure  2  [2],  Details  are  presented 
in  Annex  C. 
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Strategic  Defense  Management  Loop 
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Figure  2:  The  strategic  defense  management  loop 
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ORGANIZATION 


2.0  APPROACH 

2.1  Independent  Cost  Estimates 

SAS-076  used  the  six  major  steps  of  Figure  3  to  generate  scientifically-sound  cost  estimates  for  multiple  acqui¬ 
sition  programs  under  consideration. 


Matter  Experts 


Figure  3:  Methodology  for  the  development  of  independent  cost  estimates 


More  specifically,  for  each  program,  SAS-076: 

•  Established  Needs  with  the  Customer. 

-  Defined  and  managed  expectations  among  stakeholders  on  cost-analysis  activities,  events  and  prod¬ 
ucts  throughout  the  life  of  the  estimate. 

•  Established  a  Program  Baseline. 

-  Defined  the  program  to  include  all  technical  and  programmatic  information  required  to  generate  the 
cost  estimate. 

•  Developed  the  Baseline  Cost  Estimate. 

-  Generated  an  estimate  by  collecting  data,  selecting  methods,  building  models,  and  analyzing  risks 
and  uncertainties. 

•  Conducted  Risk  and  Uncertainty  Analysis. 
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-  Used  an  acceptable  statistical  technique  to  develop  a  probability  distribution  for  the  estimate. 

•  Verified  and  Validated  the  Cost  Estimate. 

-  Assessed  critically  the  inputs,  outputs,  and  methodology  of  the  cost  estimate  using  peer  reviews  and 
cross-checks. 

•  Presented  and  Defended  the  Estimate. 

-  Finished  documenting  the  estimate  to  produce  an  audit  trail  of  source  data,  methods,  and  results, 
and  present  the  estimate  to  principal  decision  makers  in  a  cogent,  understandable,  and  informative 
fashion. 

Since  a  “second  set  of  eyes”  is  of  paramount  importance  in  the  discipline  of  defense  cost  analysis,  subject 
matter  experts  provided  guidance  and  feedback  at  critical  review  points. 

2.2  Role  of  Life-Cycle  Cost  Analysis  in  Managing  the  Defense  Enterprise 

SAS-076  developed  the  following  templates  to  ensure  a  common  denominator  of  responses  from  NATO  and 
PfP  nations: 

•  Strategic  Framework 

-  Captures  national  practices  in  defining,  managing,  and  implementing  national  security  strategy 

•  Needs  and  Solutions 

-  Defines  the  process  for  identifying  military  needs  and  developing  and  acquiring  solutions 

•  Lexicon  and  Taxonomy 

-  Captures  national  definitions  of  "defense  capability"  and  defines  methods  that  might  be  employed 
in  grouping  capabilities 

•  Role  of  Life-Cycle  Cost  Analysis 

-  Captures  the  degree  to  which  life-cycle  cost  analysis  plays  a  role  in  planning,  acquisition,  and 
budgeting  processes  in  each  nation. 

3.0  SUMMARY  OF  RESULTS 

3.1  ICE  on  NATO  Alliance  Ground  Surveillance  System  (AGS) 

The  SAS-076  Task  Group  focused  its  work  on  generating  an  independent  cost  estimate  (ICE)  on  the  acquisition 
phase  of  the  program.  This  phase  represents  over  80%  of  the  cost  of  the  contract  to  be  awarded  to  Northrop 
Grumman.  The  ICE,  in  turn,  was  based  on: 

•  Historical  costs  of  Global  Hawk  production  (Blocks  30  and  40), 

•  An  analogy  for  the  new  radar  (Multi-Platform  Radar  Technology  Improvement  Program),  based  on  the 
AES  A  radar  on  the  F/A- 1 8  E/F  aircraft, 
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ORGANIZATION 


Y 


Figure  4:  Estimating  the  cost  for  software  development  (example) 


•  Cost  Estimating  Relationships  (CERs)  for  software  development,  as  illustrated  in  Figure  4. 

Operating  and  support  costs  for  the  in-service  portion  of  the  program  were  not  formally  proffered  although 
one  national  participant  in  the  group  did  some  rough-order-of-magnitude  estimates. 

The  Task  Group  conducted  risk  and  uncertainty  analysis  for  the  acquisition  program  using  benchmark 
coefficients  of  variation  (CV)  coupled  with  a  point  estimate  to  generate  a  cumulative  probability  distribution, 
or  5-curve,  as  shown  in  Figure  5. 

Results  of  the  analysis  were  completely  documented  with  the  ICE  delivered  to  the  NATO  AGS  Management 
Agency  (NAGSMA). 

3.2  ICE  on  Royal  Netherlands  Navy  LPDs 

The  full  ICE  was  completed  by  estimating  development  and  construction  cost  as  an  ex  post  exercise  and  O&S 
cost  as  an  ex  ante  exercise.  The  SAS-076  Task  Group  used  a  novel  application  of  known  data  mining  algorithms 
to  the  problem  of  estimating  the  cost  of  development  and  construction  for  the  Netherlands’  LPDs.  In  a  blind, 
ex  post  exercise,  the  Task  Group  set  out  to  estimate  the  costs  of  these  vessels  and  to  then  compare  the  estimates 
to  actual  costs  (the  Netherlands  Royal  Navy  withheld  the  actual  ship  costs  until  the  exercise  was  completed). 

3.2.1  Acquisition  ICE 

Technical  and  cost  data  for  ships  similar  to  the  HMS  Rotterdam  LPD  was  gathered.  A  database  of  59  ships  in 
18  classes  from  7  nations  was  compiled,  spanning  years  (commissioned)  from  1954  to  2010.  For  each  ship, 
over  a  hundred  descriptive  and  technical  ship  attributes  were  obtained,  encompassing  dimensions,  performance, 
power  generation,  lift  capacity,  armament  &  countermeasures,  sensors,  combat  &  weapon  control  systems,  etc. 


8 


RTO-TR-SAS-076 


NATO  INDEPENDENT  COST  ESTIMATING  AND  THE  ROLE  OF  LIFE 
CYCLE  COST  ANALYSIS  IN  MANAGING  THE  DEFENCE  ENTERPRISE 


Estimated  Acquisition  Cost  of  NATO  AGS 
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Estimated  Acquisition  Cost  in  Bill  ions  ofThen-Year  Euros 


Figure  5 :  Estimated  acquisition  cost  of  NATO  AGS 


Ship  development  and  production  cost  data  was  expressed  in  various  currencies  and  a  normalization  procedure 
was  required.  Two  cost  estimating  approaches  were  taken:  parametric  analysis  and  costing  by  analogy. 

Parametric  approach 

Traditional  ship  building  cost  estimating  relationships  (CERs)  are  often  mathematically  simple  or  involve  linear 
regression  analysis  (of  historical  systems  or  subsystems)  on  a  single  parameter.  This  is  often  insufficient  -  other 
cost  driving  factors  must  be  incorporated  to  develop  estimates  of  sufficient  quality  at  the  preliminary  design 
phase.  A  parametric  approach  for  ship  cost  estimation  that  incorporates  a  multitude  of  cost  driving  factors, 
while  remaining  a  top-down  approach  applicable  in  early  design  phases  of  the  procurement  cycle  was  used. 
The  M5  Model  Tree  Algorithm  combines  features  of  decision  trees  with  linear  regression  models  to  both  clas¬ 
sify  similar  ships  and  build  piece-wise  multivariate  linear  regression  models.  Figure  6  depicts  the  M5  model 
tree  applied  to  the  NATO  RTO  SAS  076  ship  data  set. 

Costing  by  Analogy 

Cost  estimation  by  analogy  is  typically  accomplished  by  forecasting  the  cost  of  a  new  system  based  on  the 
historical  cost  of  similar  or  analogous  system.  There  must  be  a  reasonable  correlation  between  the  new  and 
historical  system.  Subjectively  chosen  complexity  factors  are  often  used  to  adjust  the  analogous  system’s  cost 
to  produce  an  estimate.  The  credibility  of  the  estimate  for  the  new  system  may  be  undermined  if  the  adjustment 
factors  are  not  substantiated  -  this  is  a  key  disadvantage  of  the  traditional  analogy  method.  Flierarchical  cluster 
analysis  was  used  for  a  novel  cost  estimation  by  analogy  approach  void  of  the  subjectivity  inherent  (of  the  tra- 
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#  of  LCAC  in  Well  Deck 


LM1  LM2  Length  (m)  LM3 

<  172.y/\>  172.3 
LM4  LM5 


LM6  Range  (total  distance  in  nmi) 

<  880o/\>  8800  or  ? 
LM7  LM8 


Figure  6:  M5  model  tree  applied  to  the  NATO  RTO  SAS  076  ship  data  set 


ditional  approach)  in  quantifying  the  cost  of  the  technical  and  other  differences  between  the  historical  system 
and  the  new  system.  The  approach  also  considers  multiple  analogous  systems  rather  than  just  one.  Figure  7 
illustrating  the  arrangement  of  the  clusters  produced  by  the  hierarchical  clustering  of  ships. 


Figure  7:  Dendrogram  illustrating  the  arrangement  of  the  clusters  produced  by  the  hierarchical  clustering  of 
ships 
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Results 

The  Royal  Netherlands  Navy  revealed  the  actual  development  and  production  costs  of  HMS  Rotterdam  and 
Johan  de  Witt  LPDs  to  the  Task  Group  once  the  cost  estimates  were  established.  Figures  8  and  9  illustrate 
where  the  actual  costs  (thick  vertical  lines)  fall  with  respect  to  the  log-normal  probability  density  and  cumulative 
distribution  functions  for  the  M5  model  tree  and  hierarchical  cluster  cost  estimates  for  HMS  Rotterdam  LPD 
and  Johan  de  Witt  LPD  (cost  figures  normalized  to  a  fictitious  notional  common  currency). 


Figure  8:  Probability  density  function  (a)  and  cumulative  distribution  function  (b)  of  the  M5  model  tree  (blue) 
and  hierarchical  clustering  (red)  estimates  of  HMS  Rotterdam  LPD  cost 


(a)  Probability  density  functions  (b)  Cumulative  distribution  functions 

Figure  9:  Probability  density  function  (a)  and  cumulative  distribution  function  (b)  of  the  M5  model  tree  (blue) 
and  hierarchical  clustering  (red)  estimates  of  HMS  Johan  de  Witt  LPD  cost 


3.2.2  Operational  &  Support  ICE 

The  ICE  for  operational  &  support  (O&S)  costs  of  the  HMS  Rotterdam  LPD  ship  was  an  ex-ante  exercise  based 
on  real  organizational,  operational,  technical,  and  economic  data  gathered  from  the  Royal  Netherlands  Navy 
(RNLN). 
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Methods 

The  ICE  was  based  on  three  methods:  parametric,  analogy  and  engineering  "bottom-up"  methods.  The  para¬ 
metric  and  analogy  methods  were  applied  by  experts  from  the  United  States  Naval  Centre  for  Cost  Analysis 
(NCCA)  and  were  based  on  U.S.  ship  data  pulled  from  the  U.S.  Naval  Visibility  and  Management  of  Operat¬ 
ing  and  Support  Costs  (VAMOSC)  management  information  system.  These  methods  were  applied  on  the  U.S. 
Department  of  Defense  defined  Cost  Analysis  Improvement  Group  (CAIG)  Cost  Breakdown  Structure  (CBS). 
The  SAS-076  task  group  applied  the  engineering,  bottom-up  method  using  data  from  the  RNLN  and  the  CBS 
recommended  in  SAS-054.  The  most  important  reason  for  applying  this  detailed  method  is  that  it  highlights 
the  critical  elements  of  the  Rotterdam  LPD  and  the  logistical  organization  and  that  it  shows  the  economical 
consequences  of  the  technical  system  characteristics  over  time,  allowing  to  evaluate  the  implication  in  costs  for 
possible  system  solutions. 

Results 

Besides  the  effort  performed  by  NCCA  based  on  the  VAMOSC  data,  SAS-076  managed  to  produce  an  estimate 
for  the  O&S  costs  based  on  the  planned  operational  profile.  The  data  provided  by  the  RNLN,  however,  was 
not  sufficient  to  produce  a  reliable  cost  estimate.  Nevertheless,  using  open  source  data  SAS-076  was  able  to 
produce  an  estimate  for  the  O&S  costs.  Dividing  the  HMS  Rotterdam  ship’s  life  cycle  cost  into  acquisition 
(development  and  production)  costs  and  O&S  costs,  SAS-076  estimated  that  cost  of  O&S  over  30  years  of 
service  represents  84%  of  the  total  life  cycle  cost. 

3.3  Role  of  Life-Cycle  Cost  Analysis  in  Managing  the  Defense  Enterprise 

After  an  extensive  literature  review,  the  SAS-076  Task  Group  conducted  a  Capability  Portfolio  Analysis  con¬ 
ference  in  Paris,  France,  in  an  effort  to  learn  more  about  the  application  of  the  discipline  in  the  international 
defense  establishment.  The  group  found  that  many  existing  models  and  processes  within  NATO  fall  short  of 
the  ideal  goal  of  addressing  all  strategic  requirements  and  the  capabilities  and  costs  of  all  components  of  the 
portfolio,  large  and  small  alike. 

One  exception  was  the  Strategy-to-Tasks  Model  (STAM)  of  United  States  Special  Operations  Command 
(USSOCOM),  headquartered  at  MacDill  Air  Force  Base  in  Florida.  USSOCOM  is  the  combatant  command 
for  the  worldwide  use  of  Special  Forces  such  as  Navy  Seals  and  Delta  Operators.  STAM  does  a  good  job  of 
matching  budget  to  strategy.  And  it  uses  impressive  methodology  in  evaluating  the  warfighting  value  of  assets 
in  the  USSOCOM  portfolio.  However,  USSOCOM,  unlike  most  ministries  of  defense  within  the  Alliance,  is 
not  in  the  business  of  acquiring  multi-billion  dollar  weapons  systems  since  these  are  resourced  through  the 
Services  in  the  U.S.  DoD. 

In  a  capability  portfolio  analysis  pilot  effort  in  the  U.S.,  risk-reward  bubble  diagrams,  such  as  shown  in 
Figure  10,  proved  useful. 

For  the  templates,  the  responses  obtained  from  the  sample  of  nations  summarizes  as  below: 

•  Strategic  Framework 

-  All  nations  in  the  sample  state  general  goals  or  ambitions  in  a  national  security  document.  Some 
nations  offer  more  detail  than  others.  All  nations  have  processes  and  products  that  translate  high- 
level  objectives  into  more  concrete  policy  parameters  needed  to  build  military  forces.  Most  nations 
in  the  sample  and  many  others  within  NATO  engage  in  capability-based  planning. 

•  Needs  and  Solutions 
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Detection,  Localization,  Classification,  and  Identification  of  Sea  Mines 

ROI  =  Military  Value/Bubble  Size 
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Figure  10:  Risk-reward  bubble  diagram  (example) 


-  Most  of  the  sample  nations  described  their  process  to  identify  needs.  They  all  use  capabilities  as  a 
part  of  defense  planning  and  execute  a  gap  analysis.  All  nations  seem  to  have  alternative  solutions 
that  in  different  ways  are  prioritized  before  any  decisions  for  future  acquisition  are  made.  Scenarios 
are  for  identifying  needs  or  for  describing  strategic  frameworks  or  both.  This  also  means  that  some 
scenarios  are  not  as  detailed  as  others.  Due  to  different  governmental  organizations  in  each  country 
the  responsibility  for  working  with  needs  and  solutions  differs  between  the  nations. 

•  Lexicon  and  Taxonomy 

-  Most  nations  from  the  sample  seem  to  define  capability  in  terms  of  ability  to  achieve  a  desired 
outcome.  Regarding  components  of  the  capabilities  there  seems  to  be  a  significant  commonality. 
The  result  of  the  template  shows  that  some  nations  use  a  much  more  extensive  structure  than  others. 

•  Role  of  Life-Cycle  Cost  Analysis 

-  Of  the  seven  nations  answering  the  template  only  one  nation  uses  life-cycle  costing  both  for  identify¬ 
ing  defense  needs,  for  defense  programming  and  budgeting  and  for  acquisition  of  materiel  solutions. 
The  best  commonality  is  identified  in  the  field  of  acquisition  of  materiel  solutions. 
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ORGANIZATION 


Figure  1 1  provides  a  high-level  summary  of  the  SAS-076  analysis.  As  indicated  by  the  legend  in  Figure  11, 
colours  are  used  to  evaluate  the  degree  of  use  and  role  played  played  by  life-cycle  cost  analysis  during  the 
various  stages  of  defence  planning  and  acquisition.  Green  indicates  that  life-cycle  cost  analysis  is  usually 
performed  and  plays  a  role  in  decision  making.  Red  indicates  the  opposite — life-cycle  costing  does  not  play  a 
role. 
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U.S.  NLD  CAN  SYVE  NOR  GER  TUR 


Figure  11:  Macro  analysis  of  role  of  life-cycle  cost  analysis  in  defence  planning. 


4.0  LESSONS  IDENTIFIED 

The  goal  of  this  effort  was  to  apply  the  guideline  that  was  developed  by  SAS-054  in  multi-national  programs,  to 
gather  lessons  learned  related  to  the  guideline  and,  if  necessary,  to  adapt  or  enhance  the  guideline.  In  Section  4. 1 
main  lessons  identified  per  each  of  the  six  main  steps  in  generating  an  ICE  (as  per  Figure  3)  are  specified  with  a 
detailed  SAS-076  experience.  Section  4.2  presents  other,  more  generalized,  ICE  lessons  identified  that  enhance 
the  findings  of  SAS-054.  Section  4.3  presents  lessons  learned  from  the  SAS-076  capability  portfolio  analysis 
study. 

4.1  Main  ICE  Lessons  and  Related  SAS-076  Experience 

4.1.1  Establishing  Needs  with  Customers 

Lessons  Identified:  Obtain  stakeholder  buy-in  early  on,  generate  an  ICE  development  plan,  and  obtain  key 
signatures 


Example  SAS-076  Experience:  It  is  essential  to  obtain  consensus  if  not  unanimity  on  objectives  and  scope  of 
the  cost  estimate.  On  NATO  AGS,  key  discussions  were  required  with  major  players  to  obtain  permission  to 
generate  an  ICE  on  the  program,  to  obtain  cooperation  throughout  the  estimating  process,  and  to  publish  results. 
An  ICE  development  plan  was  generated  to  foster  common  understanding  and  to  document  agreements. 

4.1.2  Establishing  a  Program  Baseline 

Lesson  Identified:  While  the  aim  and  the  objective  of  the  study  are  fundamental  for  the  way  the  cost  analysis 
is  conducted,  cost  analysts  should  be  flexible  and  anticipate  changes  to  the  cost  boundary  during  the  cost  esti¬ 
mating  process. 


Example  SAS-076  Experience:  The  cost  boundary  changed  during  the  cost  estimating  process  of  the  O&S 
costs  of  the  Netherlands’  LPDs:  helicopters  were  at  first  not  included  but  additional  informations  from  the 
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RNLN  required  a  change.  Similarly,  originally  the  disposal  costs  were  included  in  the  Cost  Breakdown  Struc¬ 
ture,  but  the  RNLN  could  not  provide  a  strategy  for  the  disposal  of  the  ships  and  this  caused  the  removal  of 
these  costs  from  the  Cost  Breakdown  Structure. 

4.1.3  Developing  Baseline  Cost  Estimates 

Lesson  Identified:  Don’t  limit  the  scope  of  historical  data  to  use  and  be  open  to  new  costing  methods. 


Example  SAS-076  Experience:  When  SAS-076  gathered  data  for  ships  similar  to  the  Netherlands  LPD’s  (for 
the  acquisition  ICE)  the  group  did  not  limit  themselves  to  ships  very  close  to  the  Netherlands’  LPDs.  For 
example,  civilian  ships  like  icebreakers  were  also  included  because  data  related  to  these  ships  could  have  rele¬ 
vant  information  (e.g.,  same  size/different  function  and  different  size/same  function).  Similarly,  for  each  ship 
a  plethora  of  technical  and  performance  characteristics  where  gathered.  In  some  cases  data  was  unobtainable, 
but  rather  than  discarding  such  ships,  methodologies  were  proposed  that  could  handle  missing  values.  The  data 
available  determined,  in  a  large  extent,  the  choice  of  methods.  It  was  a  continuous  interaction  between  choices 
of  methods  and  computer  models  and  availability  of  data.  The  new  methodology  used,  based  on  data  mining,  is 
very  valuable  as  it  requires  only  publicly  available  technical  data  of  attributes  and  some  cost  data.  On  the  other 
hand  the  complexity  of  the  analogy  method  based  on  hierarchical  clustering  lead  to  computational  difficulties 
when  using  over  one  hundred  attributes  per  ship.  This  issue  was  dealt  with  by  applying  principal  component 
analysis  and  reducing  the  dimensionality  of  the  input  without  significant  loss  of  information. 

4.1.4  Conducting  Risk  and  Uncertainty  Analysis 

Lesson  Identified:  When  possible,  use  multiple  techniques  for  risk  and  uncertainty  analysis. 


Example  SAS-076  Experience:  The  use  of  historical  CVs  within  the  context  of  scenario-based  analysis  is 
easier  and  more  complete  than  traditional  Monte  Carlo  simulation.  All  elements  of  risk  and  captured,  and 
considerable  time  saved,  by  employing  benchmark  CVs.  Use  of  more  than  one  technique  may  be  in  order,  and 
a  best  practice.  For  example,  AGS  risks  might  be  first  estimated  using  Monte  Carlo  simulation,  with  benchmark 
CVs  then  serving  as  an  independent  validation  check  on  results. 

4.1.5  Verifying  and  Validating  Estimates 

Lesson  Identified:  Scrutinize  initial  cost  model  outputs  in  detail  and  if  necessary  determine  whether  to  change 
the  model  or  adjust  results  before  considering  omitting  historical  data. 


Example  SAS-076  Experience:  A  closer  look  at  the  Netherlands  LPD  acquisition  CER  showed  that  the  indi¬ 
vidual  linear  regression  models  were  mostly  intuitive:  the  cost  of  a  ship  increases  as  the  length  or  the  number 
of  landing  craft  supported  or  number  of  toipedo  decoy  systems  increase(s).  The  regression  models  also  predict 
a  shipbuilding  learning  curve  as  the  cost  of  constructing  a  ship  decreases  as  a  function  of  the  ship's  rank  in 
class.  However,  the  negative  coefficient  for  a  ship’s  range  (sailing  time)  in  the  regression  models  was  counter¬ 
intuitive.  It  seemed  unlikely  that  a  ship  would  cost  less  if  its  range  was  increased.  A  closer  look  at  the  SAS-076 
ship  data  explained  the  anomaly:  a  small  subset  of  the  ships  had  a  combination  of  low  cost  and  outlying  sail¬ 
ing  time  ranges.  Adjusting  the  regression  models  (by  disabling  the  particular  sailing  time  attribute)  offered  no 
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guarantee  that  the  regenerated  model  would  not  substitute  this  attribute  with  another  also  allocated  a  negative 
coefficient.  Similarly,  removing  certain  ships  from  the  data  set  provided  no  guarantees.  Rather  than  subjec¬ 
tively  diminishing  the  data  set,  SAS-076  noted  the  anomaly  and  discussing  them  as  part  of  the  results.  As  a  post 
analysis  SAS-076  substituted  the  Johan  de  Witt  sailing  range  (also  an  outlier)  with  the  median  sailing  range  of 
the  SAS-076  ship  data  set,  effectively  neutralizing  the  attribute.  This  resulted  in  a  revised  estimate  which  turned 
out  to  be  within  1%  of  the  actual  cost. 

4.1.6  Presenting  and  Defending  Estimates 

Lesson  Identified:  It  is  best  to  start  documenting  the  cost  estimate  as  it  is  developed.  Trying  to  recapture 
rationale,  judgments,  and  the  details  of  intricate  statistical  analyses  after  the  fact  is,  at  best,  extremely  difficult 
and  time  consuming. 


Example  SAS-076  Experience:  Due  to  the  nature  of  the  SAS-076  task  group,  which  met  3-4  times  a  year  for 
three  years,  it  was  necessary  to  compile  presentations  on  methodologies,  assumptions,  and  results  in  preparation 
for  each  meeting  in  order  to  facilitate  discussion  and  progress.  A  turnover  in  participants  over  the  years  also 
forced  the  group  to  regularly  document  its  findings  in  order  not  to  lose  any  information.  Although  hindered 
by  the  discontinuous  nature  of  the  meetings  (naturally  SAS-076  participants  had  national  obligations  to  attend 
to  between  meetings),  the  regular  updating  of  models  and  presentations  facilitated  report  documentation  and 
offered  a  trace  of  the  decisions  made  at  various  stages. 

4.2  Other  ICE  Lessons 

•  A  sound,  robust  life-cycle  cost  estimate  is  based  on  a  well-defined  program.  Accuracy  and  completeness 
of  the  definition  are  critical  in  producing  an  estimate  that  informs  rather  than  confuses  decisions  on  the 
allocation  of  scare  resources  within  NATO  and  PfP  nations.  The  better  the  definition,  ceteris  paribus,  the 
higher  will  be  the  quality,  flexibility,  and  usability  of  the  cost  estimate. 

•  An  ambiguously  defined  program  increases  the  use  of  assumptions,  drives  up  estimation  risk,  and  shifts 
the  burden  of  uncertainty  to  senior  decision  makers. 

•  Defining  the  program  is  not  a  one-step-and-stop  activity.  It’s  common  to  refine  the  definition  of  the 
program  throughout  the  estimation  process. 

•  Life  cycle  costing  is  a  data  driven  process,  as  the  amount,  quality  and  other  characteristics  of  the  available 
data  often  define  what  methods  and  models  can  be  applied,  what  analyses  can  be  performed,  and  hence, 
the  results  that  can  be  achieved. 

•  Robust  life-cycle  cost  analysis  requires  the  collection  of  metadata  from  myriad  sources. 

•  There  are  a  wide  variety  of  methods  and  models  available  for  conducting  risk  and  uncertainty  analysis  of 
life-cycle  cost  estimates  of  defense  acquisition  programs.  Each,  if  used  properly,  can  yield  scientifically 
sound  results.  That  said,  there’s  simply  no  substitute  for  taking  the  time  and  effort  to  understand  the 
technical  risks  and  challenges  in  developing  and  producing  sophisticated  defense  systems. 
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It’s  essential  to  convey  to  senior  leadership  the  notion  that  cost  estimates  are  uncertain,  that  acquisition 
programs  can  and  do  incur  difficulties,  and  that  the  probability  of  a  cost  estimate  becoming  reality,  when 
expressed  as  a  single  number,  is  precisely  zero. 
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•  Since  uncertainty  and  subjectivity  are  always  present  in  the  cost-estimating  process,  sound  procedures 
for  verification  and  validation  arc  essential  to  ensure  the  delivery  of  quality  products,  and,  importantly,  to 
capture  and  store  reliable  metadata  on  the  estimate  for  future  use  and  continual  process  improvement. 

•  Use  of  meaningful,  well-crafted  visual  displays  is  important  in  communicating  the  results  of  detailed  cost 
analyses  to  stakeholders. 

•  A  best  practice  is  to  start  documenting  the  cost  estimate  as  it  is  developed.  Trying  to  recapture  rationale, 
judgments,  and  the  details  of  intricate  statistical  analyses  after  the  fact  is,  at  best,  extremely  difficult  and 
time  consuming. 

•  Since  the  precise  distribution  of  the  cost  estimate  is  never  known  with  absolute  certitude,  it’s  perhaps 
helpful  to  provide  context  through  the  use  of  historical  norms.  An  example  would  be  comparing  an 
estimated  probability  distribution  for  a  current  acquisition  program  to  one  based  on  the  same  point  esti¬ 
mate  (such  as  the  median  or  mean)  but  using  a  variance  based  on  sample  CVs  from  analogous,  historical 
programs. 

4.3  Life  Cycle  Cost  Analysis  in  Managing  the  Defense  Enterprise 

Based  on  responses  from  seven  countries,  and  through  a  process  of  vigorous  discussion  and  even  heated  debate, 
SAS-076  identified  the  following  lessons  learned  with  respect  to  the  role  played  be  cost  analysis  in  managing 
the  defence  enterprise. 

1.  Engage  Early-On:  Use  cost  assessments  to  inform  the  strategic  decision  planning  process  on  what  a 
country  can  and  cannot  afford.  Canada  and  Scandinavian  countries  seem  the  best  practitioners  on  this 
score. 

2.  Budget  and  Program  to  the  Cost  Estimate:  Independent  cost  estimates  and  cost  assessments  are  valu¬ 
able  only  to  the  degree  they  are  used.  A  best  practice  is  to  generate  an  independent  life-cycle  cost  estimate 
for  every  major  weapon  system  acquisition  program  and  to  reflect  this  estimate  in  planning  and  budgeting 
systems. 

3.  Make  Cost  the  Denominator  in  Needs  and  Solutions  Analysis:  In  identifying  military  needs  and 
solutions,  capability-based  planning  is  the  gold  standard  in  NATO  today.  A  best  practice,  within  the 
context  of  capability  portfolio  analysis  as  illustrated  in  Figure  10,  is  to  compute  a  return  on  investment 
for  defence  assets  defined  as: 

Military  Value 
Life-Cycle  Costs 

It's  important  to  note  that  risk  and  balance  are  other  essential  dimensions  of  a  complete  analysis. 

4.  Move  to  the  “Left”  in  Acquisition:  Many,  if  not  all,  countries  extensively  support  the  acquisition 
process  at  key  decision  points,  called  “gates”  in  the  United  Kingdom,  milestones  in  the  United  States. 
However,  many  cost  analysts  object  to  providing  cost  estimates  early  in  acquisition  when  systems  are 
only  vaguely  defined  and  exist  only  in  design.  However,  paradoxically,  this  is  when  the  cost  estimates  are 
of  the  most  importance.  Much  more  extensive  cost-analysis  support  needs  to  be  provided  in  early  design 
phases.  Rough  order  of  magnitude  estimates  a  this  point  in  the  life  cycle  will  help  empower  informed 
decisions  on  the  allocation  of  scarce  defence  assets. 
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5.  Optimize  from  an  Alliance  Perspective:  Given  pervasive  threat  to  the  security  of  NATO  and  its  alliance 
members  across  the  broad  spectrum  of  conflict,  optimization  of  military  capability  seems  in  order.  Du¬ 
plication  of  effort,  such  as  several  countries  possessing  extensive  sea  mine  counter-measure  capability, 
might  be  avoided.  Individual  Alliance  members  might  develop  expertise  in  a  war-fighting  domain  that 
could  be  used  to  the  benefit  of  the  entire  Alliance  in  times  of  conflict.  Savings  of  hundreds  of  billions  of 
Euros  per  annum  might  be  achieved  by  managing  from  a  NATO  enterprise  perspective. 

6.  Present  Decision  Makers  with  Menu  of  Portfolios:  Senior  defense  decision  makers  express  a  strong 
demand  for  a  menu  of  acquisition  alternatives  from  which  to  choose.  Heretofore,  inordinate  attention 
has  been  devoted,  at  high  levels,  to  the  management  of  individual  acquisition  programs.  Attention  to 
acquisition  details,  at  high  levels,  might  be  better  delegated  to  subordinate  authorities  in  each  military 
department  or  acquisition  community.  Senior  decision  makers,  instead,  would  focus  on  a  complete  set  or 
portfolio  of  systems  in  any  war  fighting  or  mission  area.  A  review  of  current  and  alternative  portfolios 
would  shift  attention  of  senior  decision  makers  to  where  it  rightfully  belongs,  to  addressing  the  funda¬ 
mental,  multiple  needs  of  a  ministry  of  defense,  and  trade-off  space,  in  the  face  of  an  entire  spectrum  of 
threats. 

5.0  CONCLUSIONS 

Working  as  a  coherent  team,  the  Task  Group  was  successful  in  generating  credible,  complete,  scientifically- 
sound  independent  cost  estimates  on: 

•  HMS  Rotterdam  and  HMS  Johan  de  Witt  Landing  Platform  Docks  (LPD)  of  the  Netherlands;  and 

•  NATO’s  Alliance  Ground  Surveillance  System  (AGS). 

These  conclusions  are  offered: 

•  It  is,  in  fact,  possible  to  generate  an  ICE  on  a  major  acquisition  program  using  an  international  team  of 
dedicated  cost  analysts. 

•  It  is  difficult,  but  not  impossible,  to  share  cost  data  across  nations.  Non-disclosure  forms  need  to  be 
completed,  and  information  closely  held.  Further,  it  is  useful  if  not  essential  to  obtain  buy-in  early  on 
from  major  stakeholders.  For  example,  for  AGS,  the  SAS-076  team  generated  an  ICE  development  plan 
and  obtained  signatures  from  the  Chairman  of  the  Board  of  AGS  Management  Organization  (NAGSMO), 
the  Director  the  AGS  Management  Agency  (NAGSMA),  and  the  Director  the  SAS-076  study  group. 

•  The  SAS-054  guidelines  on  methods  and  models  for  cost  analysis  are,  in  fact,  applicable  in  a  cost  esti¬ 
mating  environment.  The  SAS-076  Task  Group  followed  these  guidelines  extensively  and  recommends 
some  enhancements, such  as: 

-  Differentiate  between  approaches  and  methods:  parametric,  analogy,  and  engineering  methods  can 
each  be  applied  in  a  top-down  or  bottom-up  approach. 

-  The  SAS-054  guidelines  for  cost  normalization  should  be  expanded  to  consider  multi-national  data 
sets  where  different  currencies  and  base  years  are  present.  The  approach  taken  by  SAS-076  in  this 
study  can  be  used  as  an  example. 
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-  Describe  the  new  methods  for  acquisition  costs  developed  by  the  group  based  on  data  mining  tech¬ 
niques. 

In  the  realm  of  managing  the  defense  enterprise,  the  SAS-076  Task  Group  offers  these  conclusions: 

•  Portfolio  analysis  techniques  can  indeed  be  employed  fruitfully  in  national  defense. 

•  Major  goals  of  portfolio  analysis  typically  include  maximizing  value  of  individual  projects,  balancing 
investments,  and  adhering  to  strategy. 

•  No  model,  of  course,  is  perfect.  That  is,  models  are  parsimonious,  plausible,  and  hopefully  informative, 
but  do  not  provide  the  ultimate  truth. 

•  Provide  senior  decision  makers  a  menu  of  investment  portfolios  from  which  to  choose. 

•  Present  joint  solution  space  in  all  investment  portfolios. 

•  Minimize  subjectivity  in  assessing  military  value. 

•  Always  assess  capabilities,  risk,  and  life-cycle  costs  of  assets.  All  three  dimensions  are  important,  and 
leaving  out  costs  will  skew  results. 

•  Newly  developed  methods  in  other  areas  than  cost  analysis  should  always  be  considered  for  application 
in  the  cost  analysis  area. 
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1.0  INTRODUCTION 

1.1  Historical  Background 

The  concept  for  an  all-weather  military  surveillance  capability  for  NATO  followed  from  Operation  Desert  Storm 
in  Kuwait  and  Iraq,  where  in  1991  the  U.S.  successfully  deployed  the  then  developmental  Joint  Surveillance 
Target  Attack  Radar  System  (JSTARS).  Unaffected  by  day /night  or  adverse  weather  conditions,  JSTARS  proved 
formidable  in  detecting  and  tracking  enemy  movements  over  vast  terrain,  while  simultaneously  disseminating 
near  real-time  data  to  U.S.  and  coalition  forces. 

In  1994,  following  a  detailed  operational  assessment  of  NATO's  military  requirements,  NATO  formalized 
Alliance  Ground  Surveillance  (AGS)  as  a  Military  Operational  Requirement  (MORS),  and  in  1995  the  North 
Atlantic  Council  (NAC)  formally  validated  the  requirement  for  a  NATO-owned  and  operated  wide-area  air-to- 
ground  surveillance  capability,  known  as  the  Alliance  Ground  Surveillance  (AGS)  system  [1], 

After  years  of  exploring  options  for  AGS  that  repeatedly  saw  the  NATO  nations  unable  to  reach  agreement 
towards  a  viable  procurement  programme;  in  September  2009,  15  of  the  28  member  nations1  signed  a  Program 
Memorandum  of  Understanding  (PMoU)  to  fund  the  developmental  phase  of  a  programme,  overseen  by  the 
NATO  Alliance  Ground  Surveillance  Management  Agency  (NAGSMA)  located  in  Brussels,  Belgium. 

1.2  AGS  Core  Overview 

Figure  1  depicts  the  general  principle  of  operation  for  the  AGS  wide-area  ground  surveillance  system.  The 
AGS  core  consists  of  a  fully  integrated  system  with  an  air  segment,  a  ground  segment  and  a  support  segment. 

AGS 

Airborne  Sensor 


Multi-User  Real-Time 
Data  Access 


Figure  1:  Principle  of  operation  of  the  AGS  core  wide-area  ground  surveillance  system 


1  [n  June  2010,  Denmark  formally  withdrew  from  the  program  as  part  of  a  move  to  cut  $500  million  from  planned  defence  spend¬ 
ing  between  2010-2014  [2],  The  14  remaining  nations  are:  Bulgaria,  Canada,  the  Czech  Republic,  Estonia,  Germany,  Italy,  Latvia, 
Lithuania,  Luxembourg,  Norway,  Romania,  Slovakia,  Slovenia,  and  the  United  States. 
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The  air  segment  will  be  based  on  the  Northrop  Grumman  RQ-4B  Global  Hawk  Block  402  high-altitude3, 
long  endurance,  unmanned  aerial  vehicle  (UAV),  equipped  with  advanced  multi-platform  radar  technology  in¬ 
sertion  program  (MP-RTIP)  for  wide-area  surveillance.  MP-RTIP  uses  active  electronically  scanned  array  radar 
to  provide  a  near-real-time  horizontally  integrated  view  of  the  battlespace,  capable  of  detecting  and  tracking 
moving  objects  throughout  the  observed  territory  and  beyond  line-of-sight,  as  well  as  providing  radar  imagery 
of  target  locations  [3,  4].  Figure  2  depicts  the  main  components  of  the  system. 

The  ground  segment  will  provide  a  direct  interface  between  the  AGS  core  system  and  a  number  of  fixed  and 
deployable  Command,  Control,  Intelligence,  Surveillance  and  Reconnaissance  (C2ISR)  systems  for  ISR  data 
exploitation  and  external  dissemination  to  multiple  deployed  and  non-deployed  operational  users. 

The  deployable  components  consist  of  four  shelter-based  Transportable  General  Ground  Stations  (TGGS), 
1 1  vehicle-based  Mobile  General  Ground  Stations  (MGGS),  and  five  Command  &  Control  Units  (UCE)  that  are 
each  self-contained  elements  with  communications  and  complete  environmental  control  for  a  crew  of  two.  Also 
included  are  15  Mobile  General  Communications  Stations  (MGCS),  which  serve  as  the  mobile  communication 
component  for  the  MGGS.  In  that  sense,  each  deployable  ground  station  must  be  fully  functional  with  low 
probability  of  being  inoperative,  be  able  to  support  NATO  operations  on  a  continuous  basis,  and  being  able  to 
provide  a  fully  integrated  capability  to  each  user  without  having  to  rely  on  external  infrastructure  [1], 

The  non-deployable  component  provides  a  centralized  ISR  service  for  high-performance  processing  at  a 
Main  Operating  Base  (MOB),  currently  located  at  Sigonella  Air  Base,  Italy.  This  Main  Operating  Station 
(MOS)  will  be  operate  continuously  in  support  of  missions,  training  and  maintenance;  and  will  maintain  the 
same  data  link  connectivity  with  the  AGS  air  platform  as  the  deployable  ground  stations  albeit  through  a  satellite 
communication  (SATCOM)  link  since  the  MOS  will  most  certainly  be  beyond  line-of-sight  range  from  the 
operational  theatre  [1]. 

The  support  elements  will  support  users  from  operative  to  tactical  levels,  and  consist  of  UAV  flight  trainers, 
operator  trainers  and  remote  work  stations.  Also  included  are  Peculiar  Support  Equipment  (PSE)  which  will 
consist  of  a  set  of  specialized  tools  for  maintenance  and  repair  of  the  MGGS  and  the  MGCC  in  the  field  [5]. 


Figure  2:  Elements  of  the  NATO  AGS  system 


2  Variants  of  Global  Hawk  aircraft  are  described  by  reference  to  ‘Block’  numbers,  e.g.,  Block  10,  Block  20,  Block  30,  etc. 

3 In  submitting  its  bid,  Northrop  Grumman  has  built  its  program  around  six  MP-RTIP  equipped  Block  40  Global  Hawks  [2].  Never¬ 
theless,  NAGSMA  questions  the  bid  and  remains  committed  to  a  program  consisting  of  eight  Unmanned  Aerial  Vehicles  (UAVs). 
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1.3  Study  Objective 

The  objective  of  this  report  is  to  present  the  ICE  for  the  NATO  AGS  team  based  on  NATO  Research  &  Technol¬ 
ogy  Organization  (RTO)  System  Analysis  and  Studies  (SAS)  task  group  (SAS-054)  guidelines  for  best  practice. 
Since  this  is  a  new  acquisition  with  contract  expected  to  be  awarded  in  201 1  and  AGS  operational  by  2014,  this 
ICE  focuses  exclusively  on  development  and  production  costs  only. 

1.4  Scope 

Per  demand  from  the  NATO  AGS  Management  Agency  (NAGSMA),  the  SAS-076  Task  Group  focused  its 
work  on  generating  an  independent  cost  estimate  (ICE)  on  the  acquisition  phase  of  the  program.  This  phase 
represents  over  80%  of  the  cost  of  the  contract  to  be  awarded  to  Northrop  Grumman. 

The  ICE,  in  turn,  was  based  on: 

•  Historical  costs  of  Global  Hawk  production  (Blocks  30  and  40), 

•  An  analogy  for  the  new  radar  (Multi-Platform  Radar  Technology  Improvement  Program),  based  on  the 
AES  A  radar  on  the  F/A- 1 8  E/F  aircraft, 

•  Cost  Estimating  Relationships  (CERs)  for  software  development. 

Operating  and  support  costs  for  the  in-service  portion  of  the  program  were  not  formally  proffered  although 
one  national  participant  in  the  group  did  some  rough-order-of-magnitude  estimates. 

The  Task  Group  conducted  risk  and  uncertainty  analysis  for  the  acquisition  program  using  benchmark 
coefficients  of  variation  (CV)  coupled  with  a  point  estimate  to  generate  a  cumulative  probability  distribution, 
or  S-curve. 

Results  of  the  analysis  were  completely  documented  with  the  ICE  delivered  to  NAGSMA. 

1.5  Outline 

After  a  detailed  introduction  into  the  AGS  system  (including  the  historical  background  and  core  overview),  and 
the  objectives  for  SAS  076  as  they  apply  to  the  NATO  AGS  portion  of  task  group. 

Section  2  presents  the  AGS  program  baseline  including  the  detailed  cost  breakdown  structure.  The  entire 
report  is  structured  along  the  lines  of  the  AGS  program  baseline  (see  Figure  3),  where  the  main  cost  drivers 
are  first  specified,  followed  by  the  detailed  cost  estimates  for  each  component  described  by  the  cost  breakdown 
structure,  and  finally  concluding  the  study  with  the  risk  and  uncertainty  analysis.  In  this  sense,  a  logical  step- 
by-step  approach  was  taken  that  could  be  applied  to  future  programs  of  this  nature. 

Section  3  presents  the  main  financial  aspects  of  the  study,  including  inflation  rates,  definitions  for  constant 
year  and  then  year  dollars,  and  the  conversions  from  Raw  Aircraft  Procurement  to  Weighted  Aircraft  Procure¬ 
ment.  Section  3  concludes  with  a  discussion  on  foreign  exchange  rates  and  the  forecasted  USD/Euro  exchange 
rate  out  to  January  2015. 

Section  4  is  the  main  analysis  portion  of  the  study.  In  this  section  detailed  cost  modelling  is  performed  on 
all  system  components  based  on  the  cost  breakdown  structure  of  Table  1.  For  most  components  where  actual 
costs  existed,  learning  curves  were  established  from  the  U.S.  Global  Hawk  RQ-4  production  line.  In  other  cases 
where  statistical  significance  of  the  regression  components  was  lacking,  mean  values  of  the  Lot  4,  5  and  6  unit 
costs  defined  the  mean  unit  cost  of  the  component. 
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Section  5  concludes  the  analysis  portion  of  the  report  with  a  detailed  risk  and  uncertainty  analysis  using 
the  Enhanced  Scenario-Based  Method  (eSBM)  for  the  major  risk  elements  of  this  study  under  three  scenarios: 
“Baseline”  (Most  likely  to  occur),  “Pessimistic”,  and  “Resource  Constrained”. 

Section  6  provides  a  conclusion  to  the  report  with  a  point  discussion  on  the  main  limitations  for  the  study. 
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ORGANIZATION 


2.0  THE  AGS  PROGRAM  BASELINE 


Developing  an  ICE  for  a  major  acquisition  programme  consists  of  a  number  of  steps,  with  checks  and  balances 
established  at  each  milestone.  Figure  3  presents  a  five-step  program  which  could  be  used  for  any  life  cycle 
cost  estimate  where,  at  each  step,  activities  are  defined  and  outputs  arc  realized.  Some  steps  could  be  com¬ 
pleted  within  a  few  months,  e.g..  Establish  Needs  with  Customer,  whereas  others  could  take  years,  e.g.,  1.3 
Develop  Cost  Estimate,  where  issues  with  gathering  and  validating  data  of  un-helded  systems,  or  developing 
an  uncertainty  profile  far  into  the  future,  hamper  the  development  of  a  viable  cost  estimate. 


Figure  3:  Methodology  for  the  NATO  AGS  independent  cost  estimate 


2.1  General  Assumptions 

The  AGS  cost  team  estimated  the  program  of  record,  as  defined  by  the  NATO  AGS  Management  Agency,  as 
it  evolved  over  the  course  of  the  study.  As  requested  by  NAGSMA,  costs  were  estimated  in  both  constant  and 
then-year  dollars  and  Euros. 

2.2  The  Cost  Breakdown  Structure  (CBS) 

The  NATO  AGS  ICE  used  a  product  oriented  breakdown  structure  according  to  the  data  available,  which  was 
fairly  limited  considering  that  the  system  is  still  in  development.  The  cost  breakdown  structure  that  the  team 
developed  consists  of  12  elements  that  define  the  cost  drivers  for  the  NATO  AGS  system.  These  can  be  seen 
listed  in  Table  1 . 
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Table  1 :  The  cost  breakdown  structure  for  NATO  AGS 


WBS  # 

Cost  Element 

WBS# 

Cost  Element 

1 

NATO  AGS  UAV  System 

1.3 

Ground/Support  Segment 

1.1 

Air  Vehicle 

1.3.1 

Hardware 

1.1.1 

Airframe 

1.3.1. 1 

Main  Operating  Station  (MOS) 

1.1.1. 1 

Wing 

1.3. 1.2 

Mobile  General  Ground  Stations  (MGGS) 

1.1. 1.2 

Fuselage 

1.3. 1.3 

Mobile  General  Communications  Stations  (MGCS) 

1.1. 1.3 

Empennage 

1.3. 1.4 

Transportable  General  Ground  Stations  (TGGS) 

1.1. 1.4 

Subsystems 

1.3. 1.5 

UAV  Command  and  Control  Elements 

1.1. 1.4.1 

Nacelle 

1.3. 1.6 

Remote  Workstations 

1.1. 1.4.2 

Fairings 

1.3. 1.7 

UAV  Flight  Trainers 

1.1. 1.4.3 

Landing  Gear  +  "Other" 

1.3. 1.8 

Ground  Station  Test  &  Integration 

1.1.2 

Propulsion 

1.3.2 

Software  Development 

1.1.3 

Communications 

1.3.2. 1 

Air  Vehicle/Payload 

1. 1.3.1 

DataLinks  Link  16  included  here?  Or  under  payload? 

1.3. 2.2 

Mission  Operations  Support 

1. 1.3.2 

Satellite  Communications 

1. 3.2.3 

Transportable  General  Ground  Stations 

1.1. 3.2.1 

Ku  Satellite  Radio 

1.3. 2.4 

Mobile  General  Ground  Stations 

1.1. 3.2.2 

Satellite  Communications  (SATCOM)  Voice 

1. 3.2.5 

Mobile  General  Communications  Stations 

1.1. 3.2.3 

International  Maritime  SATCOM 

1. 3.2.6 

CSOP 

1. 1.3.3 

UHF/VHF  Communications 

1.3. 2.7 

UAV  Command  and  Control  Elements 

1.1. 3.3.1 

UHF/VHF  Radios 

1.3. 2. 8 

System  Test  &  Integration 

1.1. 3. 3.2 

UHF  Demand  Assigned  Multiple  Access  (DAMA)  SATCOM 

1.4 

Systems  Engineering/Program  Management 

1.1.4 

Navigation/Guidance 

1.4.1 

Systems  Engineering  (SE) 

1. 1.4.1 

(2)  Global  Positioning  Systems 

1.4.2 

Program  Management  (PM) 

1.1. 4.2 

OmniStar  Differential  Global  Positioning  System  (DGPS) 

1.5 

Systems  Test  &  Evaluation 

1. 1.4.3 

IFF  Transponder/  Traffic  Alert  Collision  (TCAS-II) 

1.6 

Training 

1. 1.4.4 

Worldwide  Operations  Hardware  Suite 

1.7 

Data 

1.1.5 

Central  Computer 

1.8 

Peculiar  Support  Equipment 

1.1.6 

Auxiliary  Equipment 

1.9 

Common  Support  Equipment 

1.1.7 

Integration,  Assembly,  Test  Checkout 

1.1 

Operational/Site  Activation 

1.2 

Payloads 

1.11 

Industrial  Facilities 

1.2.1 

1.2. 1.3 

Reconnaissance 

MP-RTIP 

1.12 

Initial  Spares  and  Repair  Parts 

1.2.2 

1.2.2. 1 

1.2. 2.2 
1.2.2.3 
1.2.6 

NATO  AGS  Unique 

Electronic  Support  Measures  (ESM) 

Radar  Warning  Receiver  (RWR) 

IFF  Interrogator 

Payloads  Integration  Assembly  and  Checkout 

Add-on 

General  and  Administrative 

Facilities  Capital  Cost  of  Money 

Profit 

2.2.1  The  Air  Vehicle  Segment 

The  air  vehicle  segment  is  comprised  of: 

1.  Eight  “Off-the-shelf”  (OTS)  RQ-4B  Global  Hawk  Block  40  UAVs  made  up  of  an  airframe,  i.e.,  wings, 
fuselage,  empennage,  landing  gear  and  subsystems,  its  propulsion  system  and  navigation/guidance  sys¬ 
tems; 

2.  An  OTS  Ku-band  S  ATCOM  package  for  the  transmission  of  high  volumes  of  data  to  multiple  users  within 
a  large  area; 

3.  A  wide-band  line-of-sight  datalink  package  capable  of  simultaneous  broadcast  and  directional  operation 

[i]; 

4.  A  Link  16  datalink  package  intended  for  use  in  the  transmission  of  highly  filtered  data  to  external  users; 

5.  VHF/UHF  Saturn  radio  communications  for  integration  within  European  Airspace; 

6.  OmniSTAR  wide-area  differential  global  positioning  system  (DGPS)  using  satellite  broadcast  techniques; 

7.  Identification  Friend  or  Foe  (IFF)  interrogator. 
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2.2.2  The  Payload  Segment 

The  payload  segment  is  comprised  of: 

1.  Northrop  Grumman/Raytheon’s  new  MP-RTIP  active  electronically  scanned  array  (AES A)  technology 
for  long  range,  very  high-resolution,  simultaneous  synthetic  aperture  radar  (SAR)  and  ground  moving 
target  indicator  (GMTI)  capabilities; 

2.  Electronic  Intelligence  (ELINT)  sensors  including  emitter  detection  and  identification  through  Electronic 
Support  Measures  (ESM)  and  Radar  Warning  Receivers  (RWR). 

2.2.3  The  Ground/Support  Segment 

The  ground/support  segment  comprises  two  areas  for  costing:  one  for  hardware  and  another  for  software  de¬ 
velopment.  The  hardware  components  have  already  been  discussed  in  detail  in  section  1.2  and  consist  of: 

1.  5  OTS  Command  &  Control  Units  (Unit  Control  Elements); 

2.  1 1  Mobile  General  Ground  Stations  (MGGS); 

3.  16  Mobile  General  Communications  Stations  (MGCS); 

4.  4  Transportable  General  Ground  Stations  (TGGS); 

5.  22  Remote  Workstations; 

6.  OTS  UAV  Flight  Trainers; 

7.  Deployable  Operator  Trainers. 

Software  is  a  vital  part  of  the  air  and  ground  segments  and  its  level  of  complexity  is  usually  expressed  in 
terms  of  operational  environment  and  application  domain.  For  the  operational  environment,  software  drives  the 
avionics  for  the  air  platform  including  control,  monitoring,  communication  and  navigation.  Also,  ground-based 
mission  critical  software  for  fixed  and  mobile  sites  all  lie  within  the  operational  environment  [6]. 

Within  the  application  domain,  complex  software  algorithms  are  required  for  signal  processing  of  large 
volumes  of  data  from  command,  control  and  communications  systems.  Included  in  this  domain  is  software 
used  for  the  development,  testing  and  support  of  applications  including  trainer  software. 

Software  development  for  both  domains  consists  of  Equivalent  Source  Lines  of  Code  (ESLOC)  that  is 
typically  used  to  predict  the  amount  of  programming  that  will  be  required  in  the  development  effort,  as  well  as 
to  estimate  programming  productivity  or  maintainability  once  the  software  is  produced. 

2.2.4  Miscellaneous  Support  Elements  1.4  -  1.12 

The  remaining  cost  elements  from  Systems  Engineering/Program  Management  (1.4)  to  Initial  Spares  and  Repair 
Parts  (1.12)  provide  support  to  the  main  systems  and  subsystems.  Each  element’s  cost  is  based  on  a  percentage 
of  the  total  estimated  cost  for  the  AirVehicle  +  Payload  +  GroundSegment. 


ORGANIZATION 


A  -  8 


RTO-TR-SAS-076 


ANNEX  A  -  ESTIMATING  THE  ACQUISITION  COST  OF 
THE  ALLIANCE  GROUND  SURVEILLANCE  (AGS)  SYSTEM 


3.0  FINANCIAL  CONSIDERATIONS 

Prior  to  developing  the  AGS  cost  estimate,  we  need  to  develop  cost  models  that  reflect  all  prior  year,  current 
and  future  costs  at  the  level  of  prices  of  a  base  year,  which  in  our  case  is  2010.  Termed  constant  year  (CY) 
dollars,  they  capture  the  time  value  of  money  by  adjusting  through  compounding  and  discounting  cash  flows  to 
reflect  the  increased  value  of  money. 

Costs  in  then  year  (TY)  dollars 4  reflect  the  purchasing  power  of  the  dollar  in  the  year  the  costs  are  incurred. 
Prior  year  costs  given  in  then  year  dollars  are  the  actual  costs  incurred  in  those  years.  Future  year  costs  are  the 
projected  values  that  will  be  paid  out  in  future  years. 

3.1  Inflation 

Inflation  rates  for  AGS  design,  development,  and  production  will  likely  differ  from  values  recently  experienced 
in  the  aerospace  industries  in  the  United  States,  Canada,  and  Europe.  The  ICE  team  is  using  a  baseline  value 
of  3%  inflation  per  annum  for  outyear  projections,  weighted  according  to  the  relative  contributions  of  the  14 
NATO  countries  participating  in  the  program.  Flowever,  inflation  as  measured  by  the  consumer  price  index 
seems  to  be  accelerating  in  Europe  as  the  economic  recovery  gains  traction5.  Defense  inflation  generally  runs 
higher  than  economy-wide  figures  by  perhaps  100  to  300  basis  points  per  year,  but  follows  the  same  trend.  An 
uptick  in  rates  is  a  risk  in  the  next  few  years. 


3.2  Raw  Aircraft  Procurement 


Table  2  lists  the  U.S.  constant  year  dollars  relative  to  the  base  year  2010.  For  the  raw  aircraft  procurement 
numbers  from  FY  2002  -  FY  2010,  the  discounted  present  value  (PV)  costs  in  2010  dollars  arc  determined  by 


f=8 


pv  =  E 


F y Base 

fts(i+rj*  ’ 


(i) 


where  FVsase  is  the  future  cash  flow  in  the  base  year  2010,  r  is  the  average  inflation  rate  of  3%  per  annum  for 
the  U.S.,  Canada  and  Europe,  and  the  sum  is  over  the  9-year  period  from  FY  2010  ( t  =  0)  to  FY  2002  (t  =  8). 

For  the  raw  aircraft  procurement  (RAP)  numbers  from  FY  2010  -  FY  2020,  relative  to  the  base  year,  the 
future  value  (FV)  of  the  cash  flow  is  obtained  by 


(=10 

FV=£PVBase(l  +  r)\  (2) 

t= o 

where  FV  is  now  the  future  cash  flow  relative  to  the  constant  base  year  2010,  and  the  sum  is  over  the  1 1 -year 
period  from  FY  2010  (t  =  0)  to  FY  2020  (t  =  10). 

3.3  Weighted  Aircraft  Procurement 

The  above  calculations  are  simply  a  prerequisite  to  determining  the  estimated  TY  to  CY  dollar  conversions  for 
U.S.  procurement  costs  of  NATO  AGS.  In  the  U.S.,  Total  Obligational  Authority  (TOA)  funding  is  the  most 
accurate  reflection  of  program  spending.  It  is  the  amount  of  money  received  in  a  fiscal  year  (01  October  to 

4U.S.  terminology.  In  Canada,  TY  dollars  are  referred  to  as  budget  year  (BY)  dollars. 

5Euro  area  annual  inflation  was  2.7%  in  March  2011,  up  from  2.4%  in  February.  A  year  earlier  the  rate  was  1.6%  [7], 
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ORGANIZATION 


Table  2:  U.S.  constant  year  dollars  (base  year  highlighted) 


FY02* 

FY03 

FY04 

FY05 

FY06 

FY07 

FY08 

FY09  gg^JSJ  FY1 1 

FY12 

FY13 

FY14 

Raw  Aircraft  Procurement 

0.789 

0.813 

0.837 

0.863 

0.888 

0.915 

0.943 

0.971  1.030 

1.061 

1.093 

1.126 

Weighted  Aircraft  Procurement 

0.824 

0.849 

0.874 

0.901 

0.928 

0.955 

0.984 

1.014  1.075 

1.108 

1.141 

1.175 

“01  October  2001  to  31  September  2002 


31  September)  from  Congress  to  spend  on  defense  programs.  The  spending  profile  (outlays)  we  use  for  NATO 
AGS  is  based  on  the  time  span  of  U.S.  Navy  aircraft  procurement  outlays  of  15%,  40%,  28.6%,  10%,  4.5%,  and 
1.9%  of  TOA  funding  in  each  buy  year.  In  our  case,  we  use  a  four  year  outlay  profile  and  truncate  the  spending 
to  15%,  40%,  28%,  and  17%,  where  the  percentages  reflect  the  %  of  TOA  spending  in  the  1st  to  the  fourth  year 
respectively. 

Table  2  lists  the  weighted  aircraft  procurement  (WAP)  costs  based  on  the  four  year  spending  profile.  This 
is  a  rolling  calculation,  i.e., 


/  /  15%  40%  28%  17%  \ 

WAPt  =  1  / - 1 - 1 - 1 - , 

/  \RAP,  RAPt+ 1  RAP, +2  RAPt+iJ 


(3) 


where  the  WAP  in  year  t  is  a  function  of  the  RAP  in  year  t  and  the  next  three  future  years,  i.e.,  t  +  1 . . .  t  +  3. 


3.4  Foreign  Exchange  Rate 

The  foreign  exchange  rate  is  a  major  area  for  cost  risk  and  uncertainty.  The  NATO  AGS  contract  will  be  a  firm- 
fixed  price  direct  commercial  sale  to  Northrop  Grumman,  with  a  ceiling  price  denominated  in  2007  base-year 
euros,  but  with  much  of  the  work  done  in  the  United  States.  Converting  from  dollars  to  euros,  then,  is  a  major 
issue. 

The  prevailing  view  in  the  international  finance  community  is  that  exchange  rates  are  not  predictable,  es¬ 
pecially  at  short  horizons.  There  is  really  no  reliable  method  to  forecast  exchange  rates.  Models  for  exchange 
rate  movements  are  largely  driven  by  changes  in  macroeconomic  factors  like  unexpected  economic  or  politi¬ 
cal  events,  interest  rates,  the  pattern  of  trade  between  one  country  and  another  and  what  is  known  as  absolute 
purchasing  power  parity  (PPP)  which  holds  that  goods-market  arbitrage  will  tend  to  move  the  exchange  rate  to 
equalize  prices  between  countries. 

For  forecasting,  the  random  walk  model  remains  appealing  because  it  leads  to  smaller  forecasting  errors 
than  most  other  exchange  rate  models.  In  simple  terms,  the  exchange  rate  is,  at  any  moment  of  time,  as  likely 
to  rise  as  it  is  to  fall.  For  a  simple  random  walk,  the  best  forecast  of  tomorrow’s  rate  is  today’s  rate,  i.e., 


Xt  —  p  +X(_i  +  oSt  ,  (4) 

where  Xt  is  the  exchange  rate  at  time  t,  p  is  the  expected  rate  of  growth,  a  is  the  standard  deviation,  and  et  is  a 
random  disturbance  term.  Figure  4  shows  100  possible  sample  paths  for  the  U.S.  dollar/euro  exchange  rate  from 
January  2011,  with  the  95th  percentile  bounded  by  dashed  lines.  Although  the  exchange  rate  is  varying  wildly 
about  an  expected  value  of  1.3306  from  January  2011,  the  U.S.  currency  has  lost  13  percent  of  its  value  against 
the  euro  since  the  beginning  of  the  year.  Conversely,  the  euro  has  risen  in  value,  and  has  even  approached  the 
symbolic  $1.50  threshold.  For  this  analysis,  we  have  fixed  the  U.S. /euro  exchange  rate  at  $  1.40/euro  with  the 
variability  forming  a  large  part  of  the  risk  analysis  for  this  study. 
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Figure  4:  U.S.  dollar/euro  historical  and  forecasted  exchange  rate 
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4.0  THE  AGS  COST  ESTIMATE 

The  next,  and  probably  the  most  critical,  step  in  formulating  the  ICE  consists  of  developing  the  cost  estimate, 
which  is  often  seen  as  an  iterative  process  as  costs  are  refined  through  time  and  previously  unknown  sources 
become  available.  The  latter  becomes  particularly  acute  in  developing  the  ICE  for  a  system  that  is  still  only  in 
the  development  phase. 

The  steps  involved  include: 

1 .  Select  methods  and  models 

2.  Collect,  normalize  and  analyze  data 

3.  Develop  Cost  Estimating  Relationships  (CERs) 

4.  Develop  cost  model 

5.  Analyze  uncertainties 

Each  segment  of  the  cost  breakdown  structure  (Table  1)  will  be  analyzed  in  turn. 

4.1  The  Air  Vehicle 

Cost  Element  1.1  in  Table  1  is  the  air  vehicle.  To  cost  the  air  vehicle,  we  start  by  breaking  down  each  unit 
into  its  various  components:  wing,  fuselage,  empennage  and  subsystems,  i.e.,  nacelle,  fairings,  landing  gear, 
etc  (see  Figure  5  for  an  early  production  RQ-4  schematic).  The  cost  of  each  component  is  based  on  the  produc¬ 
tion  schedule  for  the  block  40  UAV  with  baseline  cost  adjustments  to  account  for  inflation  within  a  staggered 
production  line. 

Table  3  shows  the  Global  Hawk  low  rate  initial  production  (LRIP)6  with  phased  deliveries  as  vehicles  ad¬ 
vance  in  technologies7.  For  example.  Block  10  aircraft  (Basic  RQ-4A)  consists  of  seven  air  vehicles  with  basic 
imagery  sensors,  whereas  the  Block  40  advanced  RQ-4B  (Phase  4)  will  feature  MP-RTIP  sensor  technology, 
increased  payload,  wingspan  and  length,  although  the  range  has  been  reduced  due  to  the  increased  size  and 
payload  [8], 

We  model  air-  vehicle  costs  based  on  Global  Hawk  Block  30  and  40  actual  costs  using  learning  curves8  , 
i.e., 

Y  =  axb  ,  (5) 

where  Y  is  the  cost  associated  with  lot  midpoint  quantity  x,  the  coefficient  a  is  the  cost  associated  with  the  first 
production  unit,  and  b,  the  natural  learning  curve  slope  in  log-space,  is  always  negative  implying  that  unit  costs 
decrease  as  production  increases. 

The  lot  midpoint  is  a  single  point  that  represents  the  entire  lot,  and  not  simply  the  middle  point  of  the  lot. 
An  estimate  for  the  lot  midpoint  is  defined  by  the  equation  [9] : 

(uf  +  Ul  +  2(UfxUl)^ /a,  (6) 

6Low  rate  initial  production  (LRIP)  is  used  in  U.S.  military  procurement  programs  to  indicate  the  phase  of  initial,  small-quantity 
production  of  a  weapon  system. 

7The  acquisition  of  NATO  AGS  has  continually  slipped.  Further  delays  will  increase  then-year  dollar  and  euro  costs  due  to  inflation. 

8  A  learning  curve,  also  known  as  cost  improvement  curve  (and  probably  better  suited  to  our  analysis  since  there  is  really  no  learning 
involved),  states  that  the  cost  of  delivering  items  that  are  produced  later  in  the  project  should  be  less  than  the  cost  of  producing  similar 
items  early  in  the  project. 
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Figure  5:  Northrop  Grumman  Global  Hawk  air  vehicle  features.  Northrop  Grumman  approved  for  public 
release  August  12,  1999  (Distribution  Unlimited) 


where  Uf  and  (//,  denote  the  first  and  last  units  produced  respectively  in  that  year.  Values  for  the  midpoint  can 
be  found  in  the  last  line  of  Table  3. 

4.1.1  The  Airframe  Wing 

In  September  2002,  Triumph  Aerostructures  -  Vought  Aircraft  Division  was  selected  by  Northrop  Grumman  to 
build  an  enhanced  wing  for  the  RQ-4  Global  Hawk.  The  division  is  responsible  for  wing  fabrication,  assembly 
and  structural  testing.  Delivery  of  the  first  enhanced  wing  occurred  in  July  2005  for  LRIP  Lot  4.  Unit  costs  for 
Lots  4,  5  and  6  Block  30  and  40  were  used  together  with  the  lot  midpoints  to  establish  the  form  of  the  learning 
curve  based  on  eqn.  (5).  Table  4  lists  the  unit  costs  for  lots  4  through  6  in  thousands  of  TY  dollars  (col.  3), 
which  were  converted  to  2010  CY  dollars  using  the  WAP  rates  of  Table  2. 

Regressing  the  natural  logs  of  the  FY  2010  unit  costs  against  ln(Lot  Midpoint)  yields  a  straight  line  with 
R 2  of  0.974,  Mean  Squared  Error  of  5.069  x  10  4,  and  test  statistics  found  in  Table  5.  Converting  back  from 
log-space  yields  the  learning  curve9 


Y  =  x  (Lot  Midpoint)^^^®  .  (7) 

Equation  7  is  plotted  in  Figure  6 10  with  the  actual  Lots  4,  5  and  6  costs  highlighted  as  “Actual  Unit  Costs”. 
Knowing  the  lot  midpoints  for  AGS  production  (Table  3),  we  can  extract  the  estimated  AGS  unit  wing  costs, 
highlighted  in  red  as  “Estimated  Unit  Costs”. 

9Values  have  been  sanitized  due  to  business  sensitivity. 

1()The  Y-scale  has  been  removed  due  to  business  sensitivity. 
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Table  3:  U.S.  Global  Hawk  RQ-4  production  line 


FY02" 

FY03 

FY04 

FY05 

FY06 

FY07 

FY08 

FY09 

FY10 

FY1 1 

FY12 

FY13 

FY14 

U.S.  Global  Hawk  LRIP 

Lot  1 

Lot  2 

Lot  3 

Lot  4 

Lot  5 

Lot  6 

Lot  7 

Lot  8 

Lot  9 

Lot  10 

Lot  11 

Lot  12 

Lot  13 

Block  10  Aircraft 

Block  20  Aircraft 

Block  30  Aircraft 

3 

3 

1 

3 

3 

1 

4 

5 

2 

2 

2 

2 

3 

3 

3 

Block  40  Aircraft 

1 

3 

3 

2 

2 

2 

2 

2 

Department  of  the  Navy  BAMS 
SDD  Units4 

2 

LRIP  APN‘ 

FRF*  APN 

3 

4 

NATO  AGS 

DDQ*' 

Production 

2 

2 

2 

2 

Block  30  and  40  Yearly  Totals 

1 

5 

5 

5 

5 

6 

6 

7 

10 

11 

Block  30  and  40  Cumulative 

1 

6 

11 

16 

21 

27 

33 

40 

50 

61 

First  unit  in  lot 

1 

2 

7 

12 

17 

22 

28 

34 

41 

51 

Last  unit  in  lot 

1 

6 

11 

16 

21 

27 

33 

40 

50 

61 

Approximate  lot  midpoint 

1 

3.7 

8.9 

13.9 

18.9 

24.4 

30.4 

36.9 

45.4 

55.9 

"01  October  2001  to  31  September  2002 

'’System  development  and  demonstration  (SDD)  contract  awarded  to  Northrop  Grumman  in  April  2008  [10]. 
eAir  Procurement  Navy  (APN),  represents  the  type  of  money  or  appropriation  the  U.S.  is  using  to  buy  the  UAVs. 
''Full  rate  production  (FRPj 
^Design,  Development  &  Qualification 


Table  4:  U.S.  Global  Hawk  RQ-4  wing  costs 


FY 


Unit  Costs  Unit  Costs  Lot 
(TY  $000)  (FY10  $000)  Midpoint 


2005  4 

2006  5 

2007  6 


1 

3.7 

8.9 


Table  5:  U.S.  Global  Hawk  RQ-4  wing  regression  statistics 


Coefficient  Standard  Error 

p-value 

t-value 

Constant 

1.23  10- 

0.0009 

722.15 

ln(Lot  Midpoint) 

8.31  x  10  3 

0.0600 

-10.57 
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Figure  6:  Estimated  air  vehicle  wing  unit  costs  in  U.S.  FY  10  $M 


4.1.2  The  Airframe  Fuselage 

Global  Flawk’s  fuselage  consists  of  three  pieces:  forward-section,  mid-section  and  aft-section.  All  three  sec¬ 
tions  are  fabricated  and  mated  at  Northrop  Grumman’s  Moss  Point,  Miss.  Unmanned  Systems  Center. 

As  for  the  airframe  wing,  unit  costs  for  Lots  4,  5  and  6  Block  30  and  40  were  used  together  with  the  lot 
midpoints  to  establish  the  form  of  the  learning  curve  based  on  eqn.  (5).  Table  6  lists  the  unit  costs  for  lots  4 
through  6  in  thousands  of  TY  dollars  (col.  3),  which  were  converted  to  2010  CY  dollars  using  the  WAP  rates 
of  Table  2. 


Table  6:  U.S.  Global  Flawk  RQ-4  fuselage  costs 


FY 


Unit  Costs  Unit  Costs  Lot 
(TY  $000)  (FY10  $000)  Midpoint 


2005  4 

2006  5 

2007  6 


1 

3.7 

8.9 


Regressing  the  natural  logs  of  the  FY  2010  unit  costs  against  ln(Lot  Midpoint)  yields  a  straight  line  with 
R2  of  0.841,  Mean  Squared  Error  of  4.22  x  10  3,  and  test  statistics  found  in  Table  7.  Converting  back  from 
log-space  yields  the  learning  curve 


Y  = 


x  (Lot  Midpoint)! 


(8) 


Equation  8  is  plotted  in  Figure  7  with  the  actual  Lots  4,  5  and  6  costs  highlighted  as  “Actual  Unit  Costs”. 
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Table  7:  U.S.  Global  Hawk  RQ-4  fuselage  regression  statistics 


Coefficient 

Standard  Error 

p- value 

t-value 

Constant 

3.54xl0-2 

0.0031 

202.59 

IntLot  Midpoint) 

2.40x  10-2 

0.1564 

-3.99 

Knowing  the  lot  midpoints  for  AGS  production  (Table  3),  we  can  extract  the  estimated  AGS  unit  fuselage  costs, 
highlighted  in  red  as  “Estimated  Unit  Costs”. 


Figure  7:  Estimated  air  vehicle  fuselage  unit  costs  in  U.S.  FY  10  $M 


4.1.3  The  Airframe  Empennage 

Since  1995,  Aurora  Flight  Sciences  composite  manufacturing  plant  in  Bridgeport,  W.Va.  has  been  responsible 
for  the  fabrication  and  assembly  of  the  two-spar  cantilevered  V-tail  empennage  structure  and  nacelles  systems, 
and  since  2003  has  also  built  most  of  the  composite  parts  on  the  Global  Hawk  fuselage  [1 1], 

The  unit  costs  for  Lots  4,  5  and  6  Block  30  and  40  were  used  together  with  the  lot  midpoints  to  establish 
the  form  of  the  learning  curve  based  on  eqn.  (5).  Table  8  lists  the  unit  costs  for  lots  4  through  6  in  thousands  of 
TY  dollars  (col.  3),  which  were  converted  to  2010  CY  dollars  using  the  WAP  rates  of  Table  2. 

Regressing  the  natural  logs  of  the  FY  2010  unit  costs  against  ln(Lot  Midpoint)  yields  a  straight  line  with 
R 2  of  0.995,  Mean  Squared  Error  of  1.31  x  10  4,  and  test  statistics  found  in  Table  9.  Converting  back  from 
log-space  yields  the  learning  curve 


Y  = 


x  (Lot  Midpoint)! 


(9) 
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Table  8:  U.S.  Global  Hawk  RQ-4  empennage  costs 


FY 

Lot 

Unit  Costs 
(TY  $000) 

Unit  Costs 
(FY10  $000) 

Lot 

Midpoint 

2005 

4 

1 

2006 

5 

3.7 

2007 

6 

8.9 

Table  9:  U.S.  Global  Hawk  RQ-4  empennage  regression  statistics 


Coefficient 

Standard  Error 

p-value 

t-value 

Constant  ilBfujjK 

6.23x  10-3 

0.0006 

1111.75 

ln(Lot  Midpoint) 

4.22x  10-3 

0.0255 

-24.94 

Equation  9  is  plotted  in  Figure  8  with  the  actual  Lots  4,  5  and  6  costs  highlighted  as  “Actual  Unit  Costs”. 
Knowing  the  lot  midpoints  for  AGS  production  (Table  3),  we  can  extract  the  estimated  AGS  unit  empennage 
costs,  highlighted  in  red  as  “Estimated  Unit  Costs”. 


Figure  8:  Estimated  air  vehicle  empennage  unit  costs  in  U.S.  FY  10  $M 


4.1.4  The  Airframe  Subsystems 

Global  Hawk  subsystems  include  nacelles,  fairings,  landing  gear  and  other  miscellaneous  components.  Table 
10  lists  the  unit  costs  for  lots  4  through  6  in  thousands  of  TY  dollars  (col.  3),  which  were  converted  to  2010  CY 
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dollars  using  the  WAP  rates  of  Table  2.  Results  for  a  learning  curve  analysis  were  not  statistically  significant, 
consequently  the  subsystems  costs  were  estimated  using  the  mean  value  of  the  costs  in  lots  4  through  6. 

Table  10:  U.S.  Global  Hawk  RQ-4  subsystems  costs 


FY  Lot 


Unit  Costs 
(TY  $000) 


2005  4 

2006  5 

2007  6 


Mean  Unit  Cost  = 


Unit  Costs 
(FY10  $000) 


4.2  Global  Hawk  Propulsion 

Cost  Element  1.1.2  in  Table  1  is  the  powerplant.  With  8,600  lbs  of  thrust,  Global  Hawk  uses  the  Rolls  Royce 
AE  3007  turbofan  engine.  The  AE3007H  is  a  military  variant  of  the  commercial  engine  that  is  installed  on  air 
vehicles  such  as  the  Embraer  RJ  145,  ERJ  135,  ERJ  140,  and  Cessna  Citation  X  [12]. 

Table  1 1  lists  the  unit  costs  for  lots  4  through  6  in  thousands  of  TY  dollars  (col.  3),  which  were  converted 
to  2010  CY  dollars  using  the  WAP  rates  of  Table  2.  Results  for  a  learning  curve  analysis  were  not  statistically 
significant,  consequently  the  engine  cost  was  estimated  using  the  mean  value  of  the  costs  in  lots  4  through  6. 


Table  11:  U.S.  Global  Hawk  RQ-4  powerplant  costs 


Unit  Costs  Unit  Costs 

FY  Lot  (TY  $000)  (FY10  $000) 

2005  4 

2006  5 

2007  6 

Mean  Unit  Cost  = 

4.3  Communications 

Cost  Element  1.1.3  in  Table  1  are  communications,  which  includes  an  L3  Communications  Integrated  Com¬ 
munications  System  (ICS)  as  a  multi-link  wideband  communication  system.  The  ICS  provides  a  Common 
Data  Link  compatible,  full  duplex  wideband  air-to-ground  and  satellite  data  link,  and  redundant  full  duplex 
UHF  satellite  and/or  Line-of-Sight  (LOS)  links  for  command/control.  The  ICS  consists  of  a  Common  Airborne 
Modem  Assembly,  a  SATCOM  Radio  Frequency  Assembly  (RFA),  a  SATCOM  antenna,  a  LOS  RFA,  a  LOS 
dual-band  antenna,  two  UHF  Receiver/Transmitters,  two  UHF  Power  Amplifiers,  and  two  UHF  antennas  [13]. 
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4.3.1  Datalinks 

Table  12  lists  the  unit  costs  for  lots  4  through  6  in  thousands  of  TY  dollars  (col.  3),  which  were  converted  to 
2010  CY  dollars  using  the  WAP  rates  of  Table  2.  Results  for  a  learning  curve  analysis  were  not  statistically 
significant,  consequently  the  datalink  costs  were  estimated  using  the  mean  value  of  the  costs  in  lots  4  through 
6. 


Table  12:  U.S.  Global  Hawk  RQ-4  datalinks  costs 


FY  Lot 


Unit  Costs 
(TY  $000) 


2005  4 

2006  5 

2007  6 


Mean  Unit  Cost  = 


Unit  Costs 
(FY10  $000) 


4.3.2  Ku  Band  Satellite  Radio 

The  Global  Hawk  Mission  Control  Centre  has  data  up-  and  down-links  to  the  Global  Hawk  vehicle  directly  and 
via  the  Ku  satellite  and  the  UHF  satellite  systems. 

The  unit  costs  for  Lots  5  and  6  Block  30  and  40  were  used  together  with  the  lot  midpoints  to  establish  the 
form  of  the  learning  curve  based  on  eqn.  (5).  Table  13  lists  the  unit  costs  for  lots  5  and  6  in  thousands  of  TY 
dollars  (col.  3),  which  were  converted  to  2010  CY  dollars  using  the  WAP  rates  of  Table  2. 

Table  13:  U.S.  Global  Hawk  RQ-4  Ku  band  satellite  radio  costs 


FY 


Unit  Costs  Unit  Costs  Lot 
(TY  $000)  (FY10  $000)  Midpoint 


2006  5 

2007  6 


3.7 

8.9 


Regressing  the  natural  logs  of  the  FY  2010  unit  costs  against  ln(Lot  Midpoint)  yields  a  straight  line  with 
coefficients  6.970  for  the  Constant  and  -0.053  for  the  ln(Lot  Midpoint).  Converting  back  from  log-space  yields 
the  learning  curve 


Y  =  x  (Lot  Midpoint)^^^®  .  (10) 

Equation  10  is  plotted  in  Figure  9  with  the  actual  Lots  5  and  6  costs  highlighted  as  “Actual  Unit  Costs”. 
Knowing  the  lot  midpoints  for  AGS  production  (Table  3),  we  can  extract  the  estimated  AGS  unit  Ku  band 
satellite  radio  costs,  highlighted  in  red  as  “Estimated  Unit  Costs”. 
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Figure  9:  Estimated  Ku  band  satellite  radio  unit  costs  in  U.S.  FY  10  $M 


4.3.3  Satellite  Communications  (SATCOM)  Voice 

The  unit  costs  for  Lots  4,  5  and  6  Block  30  and  40  were  used  together  with  the  lot  midpoints  to  establish  the 
form  of  the  learning  curve  based  on  eqn.  (5).  Table  14  lists  the  unit  costs  for  lots  4  through  6  in  thousands  of 
TY  dollars  (col.  3),  which  were  converted  to  2010  CY  dollars  using  the  WAP  rates  of  Table  2. 

Table  14:  U.S.  Global  Flawk  RQ-4  SATCOM  voice  costs 


Unit  Costs  Unit  Costs  Lot 

FY  Lot  (TY$ooo)  (FY10  $000)  Midpoint 

2005  4 

1 

2006  5 

3.7 

2007  6 

8.9 

Regressing  the  natural  logs  of  the  FY  2010  unit  costs  against  ln(Lot  Midpoint)  yields  a  straight  line  with 
R 2  of  0.937,  Mean  Squared  Error  of  4.12  x  10  \  and  test  statistics  found  in  Table  15.  Converting  back  from 
log-space  yields  the  learning  curve 


Y  =  x  (Lot  Midpoint)^^^®  .  (11) 

Equation  1 1  is  plotted  in  Figure  10  with  the  actual  Lots  4,  5  and  6  costs  highlighted  as  “Actual  Unit  Costs”. 
Knowing  the  lot  midpoints  for  AGS  production  (Table  3),  we  can  extract  the  estimated  AGS  unit  SATCOM 
voice  costs,  highlighted  in  red  as  “Estimated  Unit  Costs”. 


A -20 


RTO-TR-SAS-076 


ANNEX  A  -  ESTIMATING  THE  ACQUISITION  COST  OF 
THE  ALLIANCE  GROUND  SURVEILLANCE  (AGS)  SYSTEM 


Table  15:  U.S.  Global  Hawk  RQ-4  SATCOM  voice  regression  statistics 


Coefficient 

Standard  Error 

p- value 

t-  value 

Constant 

3.50xl0-2 

0.0040 

157.39 

IntLot  Midpoint) 

2.37  xl0~2 

0.0943 

-6.70 

Figure  10:  Estimated  air  vehicle  SATCOM  voice  unit  costs  in  U.S.  FY  10  $M 


4.3.4  International  Maritime  SATCOM 

The  unit  costs  for  Lots  5  and  6  Block  30  and  40  were  used  together  with  the  lot  midpoints  to  establish  the  form 
of  the  learning  curve  based  on  eqn.  (5).  Table  16  lists  the  unit  costs  for  lots  5  and  6  in  thousands  of  TY  dollars 
(col.  3),  which  were  converted  to  2010  CY  dollars  using  the  WAP  rates  of  Table  2. 

Table  16:  U.S.  Global  Hawk  RQ-4  Maritime  SATCOM  costs 


FY 

Lot 

Unit  Costs 
(TY  $000) 

Unit  Costs 
(FY10  $000) 

Lot 

Midpoint 

2006 

5 

3.7 

2007 

6 

8.9 

Regressing  the  natural  logs  of  the  FY  2010  unit  costs  against  ln(Lot  Midpoint)  yields  a  straight  line  with 
coefficients  5.348  for  the  Constant  and  -0.191  for  the  ln(Lot  Midpoint).  Converting  back  from  log-space  yields 
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the  learning  curve 


Y  =  x  (Lot  Midpoint)^^^®  .  (12) 

Equation  12  is  plotted  in  Figure  11  with  the  actual  Lots  5  and  6  costs  highlighted  as  “Actual  Unit  Costs”. 
Knowing  the  lot  midpoints  for  AGS  production  (Table  3),  we  can  extract  the  estimated  AGS  unit  Maritime  band 
satellite  radio  costs,  highlighted  in  red  as  “Estimated  Unit  Costs”. 


Figure  11:  Estimated  Maritime  SATCOM  unit  costs  in  U.S.  FY  10  $M 


4.3.5  UHF/VHF  Communications 

The  unit  costs  for  Lots  4,  5  and  6  Block  30  and  40  were  used  together  with  the  lot  midpoints  to  establish  the 
form  of  the  learning  curve  based  on  eqn.  (5).  Table  17  lists  the  unit  costs  for  lots  4  through  6  in  thousands  of 
TY  dollars  (col.  3),  which  were  converted  to  2010  CY  dollars  using  the  WAP  rates  of  Table  2. 

Table  17:  U.S.  Global  Hawk  RQ-4  UHF/VHF  communications  costs 


Unit  Costs  Unit  Costs  Lot 

FY  Lot  (TY  $000)  (FY  10  $000)  Midpoint 

2005  4 

1 

2006  5 

3.7 

2007  6 

8.9 

Regressing  the  natural  logs  of  the  FY  2010  unit  costs  against  ln(Lot  Midpoint)  yields  a  straight  line  with 
R2  of  0.963,  Mean  Squared  Error  of  1.808  x  10  4.  and  test  statistics  found  in  Table  18.  Converting  back  from 
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log-space  yields  the  learning  curve 


Y  = 


x  (Lot  Midpoint) 


(13) 


Table  18:  U.S.  Global  Hawk  RQ-4  UHF/VHF  communications  regression  statistics 


Coefficient 

Standard  Error 

p- value 

t-value 

Constant 

7.32xl0~3 

0.0009 

695.41 

ln(Lot  Midpoint)  JHHi 

4.96  xlO’3 

0.0713 

-8.89 

Equation  13  is  plotted  in  Figure  12  with  the  actual  Lots  4,  5  and  6  costs  highlighted  as  “Actual  Unit  Costs”. 
Knowing  the  lot  midpoints  for  AGS  production  (Table  3),  we  can  extract  the  estimated  AGS  unit  UHF  voice 
costs,  highlighted  in  red  as  “Estimated  Unit  Costs”. 


Figure  12:  Estimated  air  vehicle  UHF/VHF  communication  unit  costs  in  U.S.  FY  10  $M 


4.4  Navigation/Guidance 

Cost  Element  1.1.4  in  Table  1  is  Navigation/Guidance,  which  is  via  inertial  navigation  with  integrated  Global 
Positioning  System  updates.  Costs  were  estimated  using  learning  curves  and  cost  means  where  statistical 
significance  was  lacking. 
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4.4.1  Global  Positioning  Systems  (GPS) 

The  costs  for  two  GPS  units  from  Lots  4,  5  and  6  Block  30  and  40  were  used  together  with  the  lot  midpoints  to 
establish  the  form  of  the  learning  curve  based  on  eqn.  (5).  Table  19  lists  the  two-unit  costs  for  lots  4  through  6 
in  thousands  of  TY  dollars  (col.  3),  which  were  converted  to  2010  CY  dollars  using  the  WAP  rates  of  Table  2. 

Table  19:  U.S.  Global  Hawk  RQ-4  GPS  two-unit  costs 


Unit  Costs  Unit  Costs  Lot 

FY  Lot  (TY$ooo)  (FY  10  $000)  Midpoint 

2005  4 

1 

2006  5 

3.7 

2007  6 

8.9 

Regressing  the  natural  logs  of  the  FY  2010  two-unit  costs  against  ln(Lot  Midpoint)  yields  a  straight  line 
with  R 2  of  0.999,  Mean  Squared  Error  of  1.226  x  10  6,  and  test  statistics  found  in  Table  20.  Converting  back 
from  log-space  yields  the  learning  curve 

Y  =  x  (Lot  Midpoint)^^^®  .  (14) 


Table  20:  U.S.  Global  Hawk  RQ-4  GPS  two-unit  regression  statistics 


Coefficient 

Standard  Error 

p-value 

t-value 

Constant 

6.03  xl0~4 

0.0001 

10183.08 

ln(Lot  Midpoint) 

4.09  x  10-4 

0.0082 

-77.67 

Equation  14  is  plotted  in  Figure  13  with  the  actual  Lots  4,  5  and  6  costs  highlighted  as  “Actual  two-Unit 
Costs”.  Knowing  the  lot  midpoints  for  AGS  production  (Table  3),  we  can  extract  the  estimated  AGS  two-unit 
GPS  costs,  highlighted  in  red  as  “Estimated  two-Unit  Costs”. 


4.4.2  OmniStar  Differential  Global  Positioning  System  (DGPS) 

Table  21  lists  the  unit  costs  for  lots  4  through  6  in  thousands  of  TY  dollars  (col.  3),  which  were  converted  to 
2010  CY  dollars  using  the  WAP  rates  of  Table  2.  Results  for  a  learning  curve  analysis  were  not  statistically 
significant,  consequently  the  DGPS  costs  were  estimated  using  the  mean  value  of  the  costs  in  lots  4  through  6. 


4.4.3  IFF  Transponder/Traffic  Alert  &  Collision  (TCAS-II) 

Table  22  lists  the  unit  costs  for  lots  4  through  6  in  thousands  of  TY  dollars  (col.  3),  which  were  converted  to 
2010  CY  dollars  using  the  WAP  rates  of  Table  2.  Results  for  a  learning  curve  analysis  were  not  statistically 
significant,  consequently  the  DGPS  costs  were  estimated  using  the  mean  value  of  the  costs  in  lots  4  through  6. 
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Figure  13:  Estimated  air  vehicle  GPS  two-unit  costs  in  U.S.  FY  10  $M 


Table  21:  U.S.  Global  Hawk  RQ-4  DGPS  costs 


FY 

Lot 

Unit  Costs 
(TY  $000) 

Unit  Costs 
(FY10  $000) 

2005 

4 

■ 

■ 

2006 

5 

2007 

6 

■ 

Mean  Unit  Cost  = 

Table  22:  U.S 

.  Global  Hawk  RQ-4  TCAS-II  costs 

FY 

Lot 

Unit  Costs 

Unit  Costs 

(TY  $000) 

(FY10  $000) 

2005 

4 

■ 

2006 

5 

2007 

6 

■ 

Mean  Unit  Cost  = 
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4.4.4  Worldwide  Operations  Hardware  Suite 

Table  23  lists  the  unit  costs  for  lots  4  through  6  in  thousands  of  TY  dollars  (col.  3),  which  were  converted  to 
2010  CY  dollars  using  the  WAP  rates  of  Table  2.  Results  for  a  learning  curve  analysis  were  not  statistically 
significant,  consequently  the  suite  costs  were  estimated  using  the  mean  value  of  the  costs  in  lots  4  through  6. 

Table  23:  U.S.  Global  Hawk  RQ-4  operations  hardware  suite  costs 


FY  Lot 


Unit  Costs 
(TY  $000) 


2005  4  ■ 

2006  5  H 

2007  6  ■ 

Mean  Unit  Cost  = 


Unit  Costs 
(FY10  $000) 


4.5  Miscellaneous  Air  Vehicle  Components 

Cost  Elements  1.1.5,  1.1.6  and  1.1.7  in  Table  1  refer  to  the  central  computer,  auxiliary  equipment  and  integra¬ 
tion,  assembly,  test  and  checkout  respectively. 

4.5.1  The  Central  Computer 

The  costs  for  the  central  computer  from  Lots  4,  5  and  6  Block  30  and  40  were  used  together  with  the  lot 
midpoints  to  establish  the  form  of  the  learning  curve  based  on  eqn.  (5).  Table  24  lists  the  central  computer 
costs  for  lots  4  through  6  in  thousands  of  TY  dollars  (col.  3),  which  were  converted  to  2010  CY  dollars  using 
the  WAP  rates  of  Table  2. 


Table  24:  U.S.  Global  Hawk  RQ-4  central  computer  costs 


FY 


Unit  Costs  Unit  Costs  Lot 
(TY  $000)  (FY10  $000)  Midpoint 


2005  4 

2006  5 

2007  6 


1 

3.7 

8.9 


Regressing  the  natural  logs  of  the  FY  2010  central  computer  costs  against  ln(Lot  Midpoint)  yields  a  straight 
line  with  R2  of  0.927,  Mean  Squared  Error  of  1.079  x  10  2 ,  and  test  statistics  found  in  Table  25.  Converting 
back  from  log-space  yields  the  learning  curve 

Y  =  x  (Lot  Midpoint)^^^®  .  (15) 

Equation  15  is  plotted  in  Figure  14  with  the  actual  Lots  4,  5  and  6  costs  highlighted  as  “Actual  Costs”. 
Knowing  the  lot  midpoints  for  AGS  production  (Table  3),  we  can  extract  the  estimated  AGS  unit  Computer 
costs,  highlighted  in  red  as  “Estimated  two-Unit  Costs”. 
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Table  25:  U.S.  Global  Hawk  RQ-4  central  computer  regression  statistics 


Coefficient 

Standard  Error 

p- value 

t-value 

Constant  ISSflfM 

1.79xl0~2 

0.0015 

417.37 

Inf  Lot  Midpoint) 

1.21xl0~2 

0.1019 

-6.19 

Figure  14:  Estimated  air  vehicle  computer  two-unit  costs  in  U.S.  FY  10  $M 


4.5.2  Auxiliary  Equipment 

Table  26  lists  the  unit  costs  for  lots  4  through  6  in  thousands  of  TY  dollar's  (col.  3),  which  were  converted  to 
2010  CY  dollars  using  the  WAP  rates  of  Table  2.  Results  for  a  learning  curve  analysis  were  not  statistically 
significant,  consequently  the  auxiliary  equipment  costs  were  estimated  using  the  mean  value  of  the  costs  in  lots 
4  through  6. 


Table  26:  U.S.  Global  Hawk  RQ-4  auxiliary  equipment  costs 


FY 

Lot 

Unit  Costs 

Unit  Costs 

(TY  $000) 

(FY10  $000) 

2005 

4 

2006 

5 

2007 

6 

Mean  Unit  Cost  = 
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4.5.3  Integration,  Assembly,  Test  &  Checkout  (IATC) 

Sensor  integration,  assembly  and  full  flight  testing  are  major  steps  towards  finalizing  operational  deployment 
of  the  NATO  AGS  system.  The  costs  for  the  IATC  from  Lots  4,  5  and  6  Block  30  and  40  were  used  together 
with  the  lot  midpoints  to  establish  the  form  of  the  learning  curve  based  on  eqn.  (5).  Table  27  lists  the  IATC 
costs  for  lots  4  through  6  in  thousands  of  TY  dollars  (col.  3),  which  were  converted  to  2010  CY  dollars  using 
the  WAP  rates  of  Table  2. 

Table  27:  U.S.  Global  Hawk  RQ-4  Integration,  Assembly,  Test  &  Checkout  costs 


FY 


Unit  Costs  Unit  Costs  Lot 
(TY  $000)  (FY10  $000)  Midpoint 


2005  4 

2006  5 

2007  6 


1 

3.7 

8.9 


Regressing  the  natural  logs  of  the  FY  2010  IATC  costs  against  ln(Lot  Midpoint)  yields  a  straight  line  with 
R 2  of  0.958,  Mean  Squared  Error  of  2.446  x  1 0  1 .  and  test  statistics  found  in  Table  28.  Converting  back  from 
log-space  yields  the  learning  curve 


Y  =  x  (Lot  Midpoint)^^^®  .  (16) 

Table  28:  U.S.  Global  Hawk  RQ-4  Integration,  Assembly,  Test  &  Checkout  regression  statistics 


Coefficient 

Standard  Error 

p- value 

t-value 

Constant 

2.69  xl0~2 

0.0018 

346.71 

ln(Lot  Midpoint) 

1.83xl0-2 

0.0765 

-8.28 

Equation  16  is  plotted  in  Figure  15  with  the  actual  Lots  4,  5  and  6  costs  highlighted  as  “Actual  Costs”. 
Knowing  the  lot  midpoints  for  AGS  production  (Table  3),  we  can  extract  the  estimated  AGS  IATC  costs, 
highlighted  in  red  as  “Estimated  Costs”. 

4.6  The  Payloads 

Cost  Element  1.2  in  Table  1  are  the  payloads,  which  include  MP-RTIP  radar.  Electronic  Support  Measures 
(ESM),  Radar  Warning  Receiver  (RWR),  Identification  Friend  or  Foe  (IFF)  Interrogator,  and  Payloads  Integra¬ 
tion  Assembly  and  Checkout. 

4.6.1  Multi-Platform  Radar  Technology  Insertion  Program  (MP-RTIP) 

Since  December  2000,  Raytheon  and  Northrop  Grumman  have  coordinated  the  design,  development  and  pro¬ 
duction  of  MP-RTIP.  Its  X-band  Active  Electronically  Scanned  Array  (AESA)  radar  uses  beam  steering  that 
can  couple  electronic  and  mechanical  options.  Since  costing  for  the  MP-RTIP  sensor  is  not  available  to  NATO, 
the  ICE  team  based  its  estimate  on  an  analogous  AESA  system  currently  fielded  on  a  variety  of  platforms 
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Figure  15:  Estimated  air  vehicle  Integration,  Assembly,  Test  &  Checkout  costs  in  U.S.  FY  10  $M 


(including  the  F/A-18E/F  Super  Hornet),  as  a  proxy  variable  for  complexity  and  scale.  Using  a  weight-based 
analogy,  where  it  was  assumed  that  the  higher  the  weight  the  greater  the  cost,  the  ratio  multiplication  factor  for 
AGS  MP-RTIP  (3,000  lbs)  to  AESA  (650  lbs),  was  given  as 


Ratio  = 


MP-RTIP 

AESA 


3000 

~650~ 


=  4.6 


(17) 


From  the  theory  of  learning  curves  [9],  each  time  the  quantity  of  items  produced  doubles,  the  cost  decreases 
by  a  constant  percentage  known  as  the  Learning  Curve  Slope  (LCS).  In  log-space,  equation  (5)  becomes  a 
straight  line  with  slope 

b  =  In  (LCS)  /In  (2)  ,  (18) 

where  the  term  ln(2)  arises  from  the  doubling  of  items. 

From  the  contractor  cost  data  report  for  LRIP  AESA  production,  the  total  TY  2003  price  for  eight  units 
is  The  U.S.  independent  cost  estimate  on  AESA  determined  a  learning  curve  slope  of  90%,  which 

gives  a  value  for  b  of  Equation  (5)  can  therefore  be  re-written  as 


i=  1 


(19) 


where  Y  = 


is  the  total  Lot  1  cost,  Qj  is  the  ith  unit  of  production,  and  a  is  the  first  unit  cost,  which  is 


a  = 


in  FY  2010 , 


(20) 
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where  the  TY  cost  was  converted  to  2010  CY  dollars  using  the  WAP  rates  of  Table  2. 

The  final  MP-RTIP  estimated  unit  cost  is  given  by 

Ratio  x  a  x  0.90  =  in  FY 2010  ,  (21) 

where  the  factor  of  0.90  accounts  for  the  learning  on  the  AES  A  radar  for  the  F/A-18  E/F  Super  Hornet  aircraft. 

4.7  The  Ground/Support  Segment 

The  system  architecture  and  the  configuration  of  the  NATO  AGS  ground  segment  are  especially  developed  to 
accommodate  a  network  centric  approach,  with  emphasis  on  a  local  area  network  design  and  real-time  exchange 
of  data  between  AGS  users.  Also,  the  use  of  standardization  will  ensure  system  interoperability,  not  only  with 
the  NATO  C3I  systems,  but  also  national  ISR  systems.  Through  the  AGS  ground  segment,  NATO  forces  will 
also  have  access  to  nationally  acquired  reconnaissance  and  surveillance  data  [14], 

The  following  elements  of  the  AGS  Ground  System  (GS)  are  based  on  information  from  the  NATO  Alliance 
Ground  Surveillance  Management  Agency  (NAGSMA).  They  reflect  the  assumed  numbers  to  be  purchased. 

•  Main  Operating  Station  (MOS)  ( 1  Element) 

-  Located  at  Sigonella  Air  Base  in  Italy,  the  Main  Operating  Station  (MOS),  together  with  training  and 
logistic  support  elements,  provides  the  virtual  aircrew  for  UAV  and  ground  entity  coordination  for 
multiple  ah'  vehicles  within  different  theatres  of  operations.  This  capability  provides  UAV  command 
and  control  as  well  as  information  management  with  interoperable  NATO  and  National  C2ISR 
systems. 

-  Costs  for  the  operator  and  UAV  flight  trainers  are  included  in  the  overall  costs  for  the  MOS. 

•  Mobile  General  Ground  Stations  (MGGS)  (11  Elements) 

-  Includes  1 1  all  terrain  vehicles  with  shelters. 

•  Transportable  General  Ground  Stations  (TGGS)  (4  Elements) 

-  Includes  4  transportable  shelters  with  workstations  and  communications  equipment. 

•  Mobile  General  Communications  Stations  (MGCS)  (15  Elements) 

-  1 1  all  terrain  communications  vehicles  with  SATCOM  and  wide  band  datalink  packages  are  in¬ 
cluded  in  the  costs  for  the  MGGS. 

-  4  all  terrain  communications  vehicles  with  SATCOM  and  wide  band  datalink  packages  are  included 
in  the  costs  for  the  TGGS. 

•  UAV  Command  and  Control  Elements  (UCE)  (5  Elements) 

-  The  UCE  provides  aircraft  control,  communications,  navigation  plan  modification,  and  differential 
global  positioning  system  equipment  to  support  aircraft  takeoff  and  landing,  sensor  task  schedul¬ 
ing  and  management.  Included  are  the  costs  for  five  transportable  shelters  with  workstations  and 
communications  equipment. 

•  Remote  Workstations  (22  Elements) 
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-  Costs  for  the  22  remote  workstations  are  included  in  the  overall  costs  for  the  MGGS. 

The  AGS  GS  is  partially  comparable  to  the  GS  for  the  U.S.  Global  Hawk.  The  U.S.  Standard  Ground 
System  consists  of  the  Ground  Station  Launch  and  Recovery  Element  (LRE)  and  the  Mission  Control  Element 
(MCE),  whereas  the  LRE  GS  is  quite  comparable  to  the  UCE  from  AGS,  the  elements  and  functions  from  the 
MCE  are  distributed  to  the  MOS  and  other  mobile  GS. 

Typically,  the  hardware  components  the  stations  consist  of  shelters,  controls  and  displays,  exterior  equip¬ 
ment  for  electrical  power  and  environmental  control  and  also  communication  devices  for  SATCOM  (UHF, 
Ku-Band,  Inmarsat  for  mobile  voice  and  data  communications),  tactical  common  data,  wideband  datalink  line- 
of-sight  (WBDL-LOS)  capability  and  air  traffic  control  communications. 

The  basis  for  the  estimation  of  the  AGS  Ground  Segments  has  been  generated  by  available  information 
from  U.S.  Global  Hawk  Ground  Stations.  It  is  assumed  that  there  is  a  comparable  use  of  components  as  utilized 
in  the  U.S.  GS,  e.g.,  work  stations,  radios,  and  antennas  for  the  different  communication  channels. 

4.7.1  AGS  Ground  Segment  Parametric  Cost  Estimation 

The  SAS  076  task  group  decided  to  cany  out  the  cost  estimation  for  the  Ground  Stations  via  a  parametric  cost 
estimation  model.  Parametric  estimating  is  a  technique  that  uses  validated  relationships  between  a  project’s 
known  technical,  programmatic,  and  cost  characteristics  and  known  historical  resources  consumed  during  the 
development,  manufacture,  and/or  modification  of  an  end  item.  A  number  of  parametric  techniques  exist  that 
practitioners  can  use  to  estimate  costs.  These  techniques  include  cost  estimating  relationships  (CERs)  and 
parametric  models.  Parametric  models  are  more  complex  than  CERs  because  they  incorporate  many  equations, 
ground  rules,  assumptions,  logic,  and  variables  that  describe  and  define  the  particular  situation  being  studied 
and  estimated. 

Parametric  cost  estimation  models  are  designed  so  that  they  can  conduct  cost  estimation  with  less  hardware 
information.  Therefore  they  are  an  appropriate  tool  to  estimate  the  AGS  GS.  Missing  input  values  can  be 
replaced  by  internal  default  values  or  tabular  values.  For  the  AGS  GS,  the  cost  estimation  model  “Advanced 
Cost  Estimating  Systems  (ACES)”  was  used11.  A  general  description  of  the  model  is  provided  in  Appendix  Al. 

4.7.2  Ground  Segment  Hardware 

A  work  breakdown  structure  for  the  Ground  Stations  (Table  29)  has  been  derived  via  a  top  down  approach 
from  documents  of  the  default  Global  Hawk  GS,  where  it  has  been  assumed  that  the  systems  in  the  AGS  GS 
are  the  same  as  in  the  default  Global  Hawk  GS.  This  can  be  derived  from  the  communication  channels  (Ku 
SATCOM,  INMARSAT,  UHF  SATCOM,  Common  data  link  (CDL)  line  of  sight,  UHF/VHF  voice)  as  they  are 
implemented  in  both  systems. 

The  basic  configuration  of  the  stations  is  described  in  documents  from  the  U.S.  Ground  Stations,  but  without 
any  specifications.  Input  values  for  elements  of  the  GS  have  been  compiled  by  expert  interviews,  internet 
research,  and  from  national  documents.  If  no  references  have  been  specified,  the  input  values  should  be  treated 
as  best  assumptions. 

The  following  information  is  required  for  the  cost  estimation: 

•  Physical  Characteristics  (mechanical  /  electronic  item;  weight;  volume) 

1 1  The  analysis  in  this  section  was  completed  by  Germany.  Consequently,  translation  from  German  to  English  was  required  for  input 
and  output  ACES  software  screenshots. 
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Table  29:  The  cost  breakdown  structure  for  NATO  AGS  ground  Segment 


WBS  # 

Cost  Element 

WBS# 

Cost  Element 

1.3 

AGS  Ground  Segment  (GS)  and  Support  Segment 

1.3. 1.2. 1.4.1 

Antenna  Unit 

1.3.1 

Hardware 

1.3. 1.2. 1.4.2 

Trailer 

1.3.1. 1 

Main  Operation  Support  (MOS)  [lx] 

1.3. 1.2. 1.5 

MGGS  Terrain  Communication  Equipment 

1.3.1. 1.1 

MOS  Operation  Equipment 

1.3. 1.2.1. 5.1 

MGCE  Communication  Vehicle 

1.3.1. 1.1.1 

MOS  Crew  Workstations  (30  Workstations) 

1.3. 1 .2. 1 .5.2 

MGCE  VHF/UHF  Communication  and  SATCOM 

1.3.1. 1.1.2 

MOS  Displays 

1.3. 1.2. 1.5. 3 

MGCE  HF  Communication 

1.3.1. 1.1.3 

MOS  Processing  Unit 

1.3. 1 .2. 1 .5.4 

Other  Communication  Equipment 

1.3.1. 1.1.4 

MOS  Printer 

1 .3.1.2. 1 .6 

MGGS  Miscellaneous  /  Cabling 

1.3.1. 1.2 

MOS  Sensor  Communication 

1.3.1.2.1.7 

I&T  MGGS 

1.3.1. 1.2.1 

Antenna  Unit 

1.3. 1.3 

Transportable  General  Ground  Station  (TGGS)  [4x] 

1.3.1. 1.2.2 

Ku- Antenna  Unit 

1.3. 1.3.1 

TGGS  Shelter 

1.3.1. 1.2.3 

Wideband  Modem  Assembly  (WMA) 

1.3. 1.3.2 

TGGS  Crew  Workstations  (3x) 

1.3.1. 1.2.4 

CDL  Interfacebox  (CIB) 

1.3. 1.3. 2.1 

TGGS  Processing  Unit 

1.3.1. 1.2.5 

DC  Control  /  Amp  assy 

1.3.1. 3.2.2 

TGGS  Map  Computer 

1.3.1. 1.2.6 

AC  Power  Control 

1.3. 1.3.2. 3 

TGGS  Antenna  Controller 

1.3.1. 1.2.7 

Gigabit  Switch 

1.3.1 .3.2.4 

TGGS  Monitor  (2x) 

1.3.1. 1.2.8 

Internal  Cables  and  Wiring 

1.3.1. 3.2.5 

TGGS  Printer 

1.3.1. 1.2.9 

Computer 

1.3.1. 3.2.6 

TGGS  Software 

1.3.1.1.2.10 

Cables 

1.3. 1.3.3 

TGGS  Sensor  Communication  System 

1.3.1. 1.3 

MOS  Communication  Equipment 

1.3.1.3.3.1 

TGGS  Transit  Package  No.  1 

1.3.1. 1.3.1 

MOS  VHF/UHF  Communication  and  SATCOM 

1.3. 1.3. 3.2 

TGGS  Transit  Case  No.  2 

1.3. 1.1.4 

MOS  Other  Equipment 

1.3. 1.3. 3.3 

TGGS  Antenna  Subsystem 

1.3. 1.1.5 

MOS  Training  Facility 

1.3. 1.3.4 

TGGS  Terrain  Communication  Equipment 

1.3.1. 1.5.1 

AGS  Operator  Trainer 

1.3.1. 3.4.1 

TGGS  Communication  Vehicle 

1.3.1. 1.5.2 

UAV  Flight  Trainer 

1.3.1. 3.4.2 

TGGS  VHF/UHF  Communication  and  SATCOM 

1.3.1. 1.5.3 

MOS  Training  Processing  Unit 

1.3. 1.3.4.3 

TGGS  HF  Communication 

1.3.1. 1.6 

MOB  Software 

1.3.1.3.4.4 

Other  Communication  Equipment 

1.3. 1.1.7 

MOB  Test  &  Integration 

1.3. 1.3.5 

TGGS  Environmental  Control  System 

1.3. 1.2 

Mobile  General  Ground  Station  (MGGS)  I  llx] 

1.3. 1.3.6 

TGGS  Electrical  System 

1.3. 1.2.1 

MGGS  All  Terrain  Vehicle  /w  Shelter 

1.3. 1.3.7 

TGGS  Miscellaneous  /  Cabling 

1.3. 1.2. 1.1 

MGGS  Terrain  Vehicle 

1.3. 1.3. 8 

I&T  TGSS 

1.3. 1.2. 1.2 

MGGS  Shelter 

1.3. 1.4 

UAV  Command  &  Control  Element  (UCE)  [5x] 

1.3. 1.2. 1.2.1 

MGGS  Container  (protected) 

1.3. 1.4.1 

UCE  Shelter 

1.3.1.2.1.2.2 

MGGS  Crew  Workstations  (5  incl.  2  Remoteable  Workstations) 

1.3. 1.4.1. 1 

UCE  Container  (protected) 

1.3. 1.2. 1.2. 3 

MGGS  Sensor  Communication  (indoor  unit) 

1.3. 1.4. 1.2 

UCE  Crew  Workstations  (2x) 

1. 3.1.2. 1.2.4 

MGGS  Environmental  Control  System 

1.3. 1.4. 1.3 

UCE  and  Communication  Equipment 

1.3. 1.2. 1.2.5 

MGGS  Software 

1.3. 1.4. 1.4 

UCE  Software 

1.3. 1.2. 1.3 

MGGS  Electrical  System 

1.3. 1.4.2 

UCE  Electrical  System 

1.3. 1.2.1. 3.1 

Power  Supply  Generator 

1.3. 1.4.2. 1 

Power  Supply  Generator 

1.3. 1.2. 1.3.2 

Power  Distribution  Unit 

1.3.1 .4.2.2 

Power  Distribution  Unit 

1.3. 1.2. 1.3. 3 

Trailer 

1.3. 1.4.3 

UCE  Int  &  Test 

1.3. 1.2. 1.4 

MGGS  Antenna  Subsystem  (outdoor  unit) 

1.3. 1.5 

GS  Test  &  Integration 

•  Quantity  (Production  Quantity,  Lot  Quantity) 

•  Quality  (operating  environment  factor,  Technology  difficulty  Index  of  the  mechanical  or  electronics  por¬ 
tions). 

•  Economic  (currency,  escalation,  Year  of  Economics,  Country  of  Production,  G&A  Profit,  Cost  of  money) 

•  Development  (Engineering  Difficulty;  quantity  of  prototypes) 

•  System  relationship  (integration  complexity) 

4.7.3  Sample  Calculation:  YHF/UHF  Antenna 

To  understand  how  ACES  was  used  in  this  context,  we  present  the  case  for  the  VHF/UHF  ground  station 
antenna.  The  manufacturer  is  unknown  at  this  stage,  however  the  Harris  RF-9070  is  a  militarized,  vertically 
polarized,  omnidirectional  transportable  antenna  designed  for  ground-to-air-  communication  for  use  in  other 
UAVs  [15],  and  as  such  was  used  as  a  proxy  for  the  AGS  ground  stations. 

Figure  16  provides  the  technical  specifications  for  the  RF-9070.  From  the  parameters,  input  values  for 
the  ACES  estimation  tool  were  generated.  In  this  case,  the  weight  and  type  of  the  antenna  is  linked  with 
inherent  tool  values  (e.g.,  production  complexity,  integration  complexity,  operational  environment)  as  shown  in 
Figure  17.  The  tool  then  generates  the  probable  production  costs,  and  on  higher  integration  levels,  the  costs  for 
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integration  and  test  (see  Figure  18  and  Appendix  A2  for  translation  from  German  to  English).  Generally  the 
estimation  is  based  on  the  assumption  that  military  off-the-shelf  components  will  be  used. 

The  final  estimated  cost  for  the  VHF/UHF  antenna  can  be  found  in  Figure  1 8  in  the  row  labelled,  in  German, 
“Gesamtkosten”  (Total  Sum)  (See  Appendix  A2  for  definitions).  These  costs  include,  in  2009  USD,  $27,415 
for  development,  $1 15,991  for  production,  for  a  total  cost  of  $143,406  for  1 1  units,  or  $13,037  per  unit. 


Specifications  for  the  RF-9O70 


Electrical  Mechanical 

■  Frequency  Range  ■  RF  Connector 

1 00  to  400  MHz  (usable  90  to  470  MHz)  Female,  Type  N 


■  RF  Power  Capacity 

400  watts 

■  Input  Impedance 

50  ohm.  nominal 

■  Radiation  Pattern 

Omnidirectional 

■  Polarization 

Vertical 

■  VSWR 

S2.1  typical.  2.5:1  maximum 

■  Gain 

>2  dBi 

Environmental 

■  Temperature 

Operating:  -40'  to  +55'  C  (-40'  to  +1 31'  F)  Storage: 
-6(7  to  +7CC  (-76'  to  +1  S3'  F) 

■  Relative  Humidity 

0  to  100% 


■  Deployed  Dimensions 

76H  x  60D  in  (193H  x  152D  cm) 

■  Finish 

E512  per  MIL-F-14072 

■  Weight 

20  lb  (9.1  kg) 

■  Mounting  Interface 

1  _3  inch  (3.3cm)  threaded  hole  in  center  hub 

■  Installation  time 

10  minutes.  2  people 

Options 

■  RF-9071  Mounting  Mast  —  A  6.5-foot  (2-m)  length  of 
1  3-inch  (3.3-cm)  diameter  mast  complete  with 
mounting  damps.  The  mast  is  threaded  at  one  end 
with  an  1 1-1/2  NPT  thread  to  match  the  antenna 
hub. 


Figure  16:  Flarris  RF-9070  parameters 


RTO-TR-SAS-076 


A -33 


ANNEX  A  -  ESTIMATING  THE  ACQUISITION  COST  OF 
THE  ALLIANCE  GROUND  SURVEILLANCE  (AGS)  SYSTEM 


ORGANIZATION 


Figure  17:  Harris  RF-9070  input  to  ACES 
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Output  Screen  -  UHF  Antenna  /  $(1209)  /  Skin  Type  1 
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Figure  18:  Harris  RF-9070  output  from  ACES 
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5.0  RISK  AND  UNCERTAINTY  ANALYSIS  FOR  NATO  AGS 

5.1  Background 

The  SAS-076  Task  Group  used  Enhanced  Scenario-Based  Method  (eSBM)  [16]  to  perform  risk  and  uncertainty 
cost  analysis  for  the  AGS  acquisition.  To  conduct  eSBM,  the  Task  Group  needed  to 

•  Generate  a  point  estimate  for  acquisition  costs, 

•  Identify  the  position  of  the  point  estimate  on  the  S-Curve, 

•  Identify  and  analyze  major  elements  of  risk  and  uncertainty, 

•  Select  an  appropriate  Coefficient  of  Variation  (CV), 

•  Develop  scenarios,  and 

•  Combine  these  components  into  an  integrated  whole. 

5.2  Point  Estimate  and  Position  on  S-Curve 

The  Task  Group  employed  a  number  of  techniques  to  estimate  the  costs  of  AGS  system.  These  included 
learning  curves,  averages  of  historical  data,  CERs,  and  analogies.  In  many  cases,  cross  checks  were  developed 
based  on  German  experience  with  Eurohawk.  Since  it’s  necessary  in  eSBM  to  anchor  a  cost  estimate  to  a 
point  on  a  cumulative  probability  distribution,  baseline  costs  were  generated,  by  design,  at  the  median,  or  the 
50th  percentile.  Another  choice  could  have  been  the  mean.  Generally,  there’s  flexibility  in  choosing  either,  or 
perhaps  a  point  in-between. 

In  the  case  of  AGS,  many  items  in  the  Cost  Element  Structure  (Table  1),  were  estimated  using  unit  learning 
curves  or  power-function  CERs  with  a  multiplicative  random  error  term  (e.g.,  Y  =  aQ^ee,  where  Y  =  unit  cost, 
Q  =  lot-midpoint  quantity,  a  and  /3  are  parameters,  i.e.,  first  unit  cost  on  the  learning  curve  (Tl)  and  elasticity 
respectively,  and  e  =  V(/./ .  a2)  is  the  error  distribution,  where  we  assume  the  errors  are  normally  distributed 
with  mean  ji  and  standard  deviation  a. 

Examples  include  the  wing,  fuselage,  and  empennage  of  the  UAV,  and  final  assembly,  integration,  test 
and  check  out  (see  section  4.1).  In  these  cases,  inserting  a  value  of  an  explanatory  variable  into  the  equation 
yields  an  estimated  median  rather  than  a  mean  value  [17].  In  other  cases,  such  as  for  software  development 
where  representatives  from  several  participating  nations  each  generated  a  cost  estimate  independently,  a  middle 
value  (median)  was  selected  as  the  baseline.  Moreover,  the  CERs  employed  in  producing  the  middle  value 
were  themselves  median-yielding  power-function  equations.  In  other  cases  where  costs  appeared  normally 
distributed,  the  choice  of  median  or  mean  was  a  moot  point  since  the  values  were  equal.  Examples  include 
systems  engineering  and  program  management,  initial  spares,  and  support  equipment. 

5.3  Risk  Elements 

Next,  the  Task  Group  identified  these  major  areas  of  cost  risk  and  uncertainty12: 

12  Areas  of  risk  were  identified  based  on  general  knowledge  of  defense  acquisition  programs  and  the  discipline  of  international 
economics,  site  visits  to  Northrop  Grumman's  International  Program  Office  in  Florida  and  to  NATO’s  AGS  Management  Agency  in 
Brussels,  and  on  meetings  with  other  key  AGS  acquisition  officials,  including  the  Chairman  of  the  Board  of  Directors  of  the  NATO 
AGS  Management  Organization. 
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•  Exchange  Rate:  The  AGS  contract  will  be  a  firm-fixed  price  direct  commercial  sale  to  Northrop 

Grumman,  with  a  ceiling  price  denominated  in  2007  base-year  euros,  but  with  much 
of  the  work  done  in  the  United  States.  Converting  from  dollars  to  euros,  then,  is  a 
major  issue.  Unfortunately,  currency  exchange  rates  are  notoriously  difficult  if  not 
impossible  to  predict  accurately  and  consistently.  The  projections  of  Figure  4,  using 
random  walk  theory,  don’t  exactly  inspire  confidence  in  anyone’s  ability  to  hone  in 
on  the  future  value  of  the  $/€  exchange  rate. 

Since  its  introduction  roughly  a  decade  ago,  the  value  of  the  euro  has  varied  from  a 
low  of  $0.83/€  in  2000  to  a  peak  of  $1.60/€  in  2008,  a  swing  of  93%.  More  recently, 
during  the  height  of  the  Greek  credit  crisis  in  early  2010,  the  euro  fell  to  $1.1 8/€.  It 
then  returned  to  pre-crisis  levels  only  to  fall  once  again  with  the  Irish  debt  crisis. 

•  Inflation:  Inflation  rates  for  AGS  design,  development  and  production  will  likely  differ  from 

values  recently  experienced  in  the  aerospace  industries  in  the  United  States,  Canada, 
and  Europe.  The  ICE  team  used  a  baseline  value  of  3%  inflation  per  annum  for  out- 
year  projections,  weighted  according  to  the  relative  contributions  of  the  14  NATO 
countries  participating  in  the  program.  However,  as  Figure  19  shows,  inflation 
as  measured  by  the  consumer  price  index,  seems  to  be  accelerating  in  Europe  as 
the  economic  recovery  gains  traction.  Defense  inflation  generally  runs  higher  than 
economy- wide  figures  by  perhaps  100  to  300  basis  points  per  year,  but  follows  the 
same  trend.  An  increase  in  rates  is  a  risk  in  the  next  few  years. 


Figure  19:  Euro  area  inflation  rate  (actual  change  on  consumer  price  index).  Source:  TradingEconomics.com 

•  Schedule:  The  acquisition  of  NATO  AGS  has  continually  slipped.  Further  delays  will  increase 

then-year  dollar  and  euro  costs  due  to  inflation. 

•  Software:  European  participants  in  the  AGS  program,  and  Canada,  are  responsible  for  ground 

segment  design,  development,  and  build.  Elements  of  the  ground  segment  include 
several  types  of  ground  vehicles,  command  and  control  units,  training  equipment,  and 
an  extensive  software  development  effort.  The  baseline  count  of  equivalent  source 
lines  of  code  (ESLOC)  is  unusually  large  from  a  U.S.  perspective13,  and  includes  no 
factor  for  growth.  However,  as  Figure  20  shows,  code  growth  on  defense  software 
development  projects  has  been  the  norm. 
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ORGANIZATION 


Median  CGF=  1.14 
Mean  CGF  =  1.73 
CV  =  95% 
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ESLOC Count  Growth  Factor  (Final/Estimate);  Bin  Endpoints 


Figure  20:  Growth  count  in  estimated  lines  of  code 


•  Software  (cont’d):  In  the  case  of  AGS,  software  will  be  developed  in  many  different  countries  by  many 
different  companies,  possibly  using  different  computer  languages  and  even  operating 
systems.  Levels  of  Capability  Maturity  Model  Integration  (CMMI)  certification  vary 
among  vendors.  Integration  of  software  modules,  hardware  with  software,  and  AGS 
with  other  ISR  assets  such  as  ASTOR  from  the  U.K.  and  Global  Hawk  from  the 
U.S.  will  all  be  required.  Configuration  management  and  software  integration  will 
be  major  issues.  The  AGS  ICE  team  uses  the  average  historical  growth  in  ESLOC 
count  as  a  proxy  variable  for  risk  for  the  entire  software  development  effort. 


•  Radar:  The  MP-RTIP  payload  uses  advanced  electronically  scanned  array  (AESA)  technol¬ 

ogy  currently  employed  on  the  F/A-18,  among  other  platforms.  However,  the  MP- 
RTIP  development  program  has  experienced  significant  cost  and  schedule  growth 
which  might  translate  into  higher  unit  production  costs. 


•  Int’l.  Participation:  14  NATO  countries  intend  to  fund  the  acquisition  of  AGS.  In  return,  each  demands 
a  fair  share  of  the  work.  NATO  intends  to  award  a  contract  to  Northrop  Grumman 
Integrated  Systems  Sector  International,  Inc  (NGISSI)  who  will  have  total  system 
performance  responsibility  (TSPR)  and  will  subcontract  work  to  vendors  in  various 
countries,  with  some  of  the  potential  firms  shown  in  Figure  21.  Developing  and 
producing  hardware  and  software  under  this  work-share  constraint  in  such  a  multi¬ 
cultural,  geographically-dispersed  environment  runs  the  risk  of  introducing  ineffi¬ 
ciencies  into  the  program. 
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Figure  21:  International  participation 


•  Affordability:  Fitting  the  desired  numbers  and  capabilities  of  UAVs  and  ground-segment  elements 

under  the  NATO’s  ceiling  price  remains  a  challenge.  A  few  years  ago  NATO  and 
industry  envisioned  a  mixed  fleet  of  manned  and  unmanned  air  vehicles  to  perform 
the  ground  surveillance  mission.  Flowever,  that  option  was  scrapped  because  of  cost. 
Affordability  continues  to  be  an  issue  as  many  European  nations  struggle  to  reduce 
budget  deficits  and  national  debt.  Denmark,  for  example,  announced  its  intention  in 
2010  to  withdraw  from  the  program  because  of  budget  tightening. 


5.4  Selection  of  CV 

With  a  point  estimate  generated  and  major  elements  of  risk  identified,  the  Task  Group  needed  a  CV  to  imple¬ 
ment  eSBM.  NATO,  it’s  important  to  note,  uses  the  Phased  Armaments  Programming  System  (PAPS)  as  its 
acquisition  framework.  In  PAPS,  AGS  is  near  what  we  call  a  Milestone  (MS)  B  decision  in  the  United  States. 
Further,  although  AGS  is  an  Alliance  rather  than  a  U.S.  DoD  acquisition  program,  the  prime  contractor  will  be 
Northrop  Grumman,  and  close  to  two-thirds  of  the  costs  will  be  incurred  in  the  U.S.  The  use  of  benchmark  CV 
data  from  the  U.S.,  then,  seems  appropriate,  especially  since  none  is  available  for  NATO  acquisitions. 

The  Task  Group  selected  a  MS  B  CV  of  51%  based  on  the  full  sample  of  data  for  quantity-adjusted  then- 
year  dollar  acquisition  outcomes.  That  is,  the  Task  Group  regarded  the  quantity  of  UAVs  and  ground  vehicles 
as  exogenous  but  the  rate  of  inflation  as  random,  for  purposes  of  generating  an  S-curve.  Given  affordability  and 
exchange  rate  issues,  the  massive  software  development  effort,  and  extensive  international  participation  with 
each  country  demanding  "noble  work,"  the  Task  Group  judged  the  higher-end  of  Milestone  B  CV  values  to  be 
appropriate. 
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5.5  Scenarios 

The  Task  Group  bundled  risks  and  uncertainties  into  three  scenarios:  baseline,  pessimistic,  and  resource- 
constrained.  The  baseline  scenario  used  “likely-to-occur”  values  or  assumptions  as  shown  in  Figure  2214: 

•  An  exchange  rate  of  $1.35  per  euro; 

•  An  inflation  rate  of  3%  per  annum,  based  on  best-estimates  of  inflation  in  the  aerospace  industries  of  the 
U.S.,  Canada,  and  Europe,  weighted  according  to  program  costs  to-be-incurred  by  each; 

•  Procurement  of  8  UAVs  and  15  ground-segment  vehicles  in  accordance  with  NATO’s  AGS  Program 
Memorandum  of  Understanding; 

•  Contract  award  in  late  FY201 1 ; 

•  91%  learning  on  MP-RTIP,  based  on  analogy  to  the  Advanced  Electronically  Scanned  Array  (AES A) 
radar  on  the  F/A-18  E/F  aircraft;  and 

•  Built-in  redundancies  or  contingencies  for  operating  in  a  NATO  acquisition  environment. 


Elements 

Baseline 

Pessimistic 

Constrained 

Exchange  rate 

$1.35  perC 

&.xx  per€ 

Sx.xx  per€ 

1  nflation  rate 

3.00% 

x% 

x% 

slip  in  schedule 

Qu  an  tities/Sch  ed  ule 

FY11  FY12  FY13  FY14  FY15< 

FTll  FY12  FY13  FYi4 

FY11  FY12  FY13  FY14  FY15 

UAVs 

2  2  2  2  0 

Ground  Stations 

/  change  in  quantities  \ 

Transportable 

11110 

\  and  schedule  / 

Mobile 

2  3  3  3  0 

ESUOC  Count 

No  growth 

x%  increase 

No  growth 

Radar 

91%  learning 

x%  learning 

91%  learning 

Int'l  Participation 

Built-in  redundancy 

x% delta  to  SE/PM 

Built-in  redundancy 

Affordability 

Unconstrained 

Unconstrained 

Constrained 

Figure  22:  Task  Group  scenarios 

In  keeping  with  the  tenants  of  eSBM,  the  non-baseline  scenarios  did  not  represent  extreme  cases,  but  rather  a 
set  of  conditions  that  could  easily  occur  in  the  future.  For  example,  the  worst  imaginable  case  for  the  U.S.  dollar 
or  the  Euro  might  be  a  severe  devaluation  of  either15.  On  the  other  hand,  modest  appreciation  or  depreciation 
of  the  dollar  or  Euro  is  certainly  plausible,  depending  upon  circumstances. 

l4Data  that  might  be  construed  as  business  sensitive  are  omitted  from  the  display. 

15Pundits  in  Europe  and  the  U.S.  speculated  in  February  2010,  for  example,  that  the  European  Union,  and  the  Euro,  might  not 
survive.  Other  pundits  project  financial  Armageddon  for  the  U.S.  because  of  our  enormous  direct  national  debt  ($14  trillion)  and 
unfunded  liabilities  for  Social  Security,  Medicare,  and  other  entitlement  programs  (another  $50  trillion). 
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5.6  S-Curve 

With  a  point  estimate,  CV,  and  anchored  position  on  the  S-curve  in  hand,  Figure  23  shows  eSBM  results.  Values 
on  the  X-axis  could  be  construed  as  somewhat  business-sensitive,  prior  to  contract  award,  and  are  therefore  not 
displayed. 
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Figure  23:  Estimated  acquisition  cost  of  NATO  AGS  ( Cost  values  are  not  displayed  due  to  business  sensitivity ) 

Several  points  merit  emphasis: 

•  Not  all  areas  of  risk  and  uncertainty  are  captured  in  the  scenarios,  nor  do  they  need  to  be.  That’s  the  great 
advantage  of  employing  a  historically-based  CV  in  computing  an  S-curve.  The  distribution  implicitly 
captures  the  totality  of  events  which  will  influence  the  cost  of  the  acquisition.  A  maintained  hypothesis, 
of  course,  is  that  historical  variances  will  continue  to  hold. 

•  The  scenarios  reflect  cost  estimates  for  the  events  or  assumptions  they  portray.  In  supporting  the  NATO 
AGS  Management  Agency  (NAGSMA)  in  Brussels,  the  Task  Group  estimated  and  presented  costs  of 
each  element  of  risk  (e.g.,  the  cost  impact  of  a  50%  increase  in  ESLOC  count)  within  a  scenario.  The 
associated  probabilities  for  the  scenarios  were  simply  read  from  the  S-curve. 

•  The  resource-constrained  scenario  was  not  displayed  on  the  S-curve  of  Figure  6,  and  was  instead  regarded 
as  a  distinct,  separate  what-if  drill.  Estimating  the  risk  and  uncertainty  of  changes  in  acquisition  quantity 
is  generally  beyond  the  purview  of  cost  analysts.  The  changes  often  result  from  Congressional  action, 
which  can  be  hard  to  predict.  The  historical  CVs  presented  here  are  all  adjusted  for  quantity  variation;  if 
they  weren’t,  they’d  be  far  higher. 

•  The  probability  of  NATO  AGS  acquisition  costs  increasing  from  the  baseline  to  the  pessimistic  scenario 
is  roughly  one  in  four  or  five.  Identification  and  quantification  of  this  possibility  can  form  the  basis  of 
risk-management  planning  by  both  the  program  office  (NAGSMA)  and  the  prime  contractor. 
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5.7  S-Curve  Excursion 

To  demonstrate  the  consequences  of  using  an  inaccurate  CV,  Figure  24  shows  a  second  S-curve  based  on  a 
hypothetical  value  of  10%. 
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Figure  24:  Estimated  acquisition  cost  of  NATO  AGS  ( Cost  values  are  not  displayed  due  to  business  sensitivity ) 

If  the  10%  value  had  been  used,  the  probability  of  costs  reaching  or  exceeding  the  value  of  the  pessimistic 
scenario  would  be  calculated  as  only  five  in  one  million !  Yet,  to  repeat,  the  parameters  used  in  this  scenario  are 
realistic,  modest  deltas  to  the  baseline.  They  include  what  the  Task  Group  regarded  as  a  very  slight  change  in  the 
exchange  rate,  a  modest  increase  in  the  rate  of  inflation,  use  of  a  mean  growth  factor  for  number  of  equivalent 
source  lines  of  code,  and  a  percentage  increase  in  the  cost  of  system  engineering/program  management  due  to 
international  participation  that  at  least  one  member  of  the  team  thought  was  appropriate  only  if  two  rather  than 
14  nations  were  involved  in  AGS  design,  development,  and  build.  In  short,  the  steep  S-curve  would  deceive 
stakeholders  into  believing  that  risk  was  slight  and  the  probability  of  significant  cost  growth  minimal. 

Specification  and  adoption  of  a  pessimistic  scenario  in  eSBM,  which  can  easily  be  adopted  in  Monte  Carlo 
methodology,  too,  has  the  virtue  of  providing  the  means  with  which  to  perform  a  sanity  check  on  the  value  of 
the  CV.  If  the  resulting  S-curve  is  too  steep  to  be  plausible,  then  underlying  risk  and  uncertainty  distributions 
for  major  cost  elements  must  not  have  been  calculated  correctly.  Chances  are  that  at  least  one  was  too  narrow. 

Finally,  since  the  mean  of  a  lognormal  distribution  involves  a  variance  term,  use  of  the  steeper  S-curve 
would  have  resulted  in  a  lower  value  of  expected  cost. 


A -42 


RTO-TR-SAS-076 


ANNEX  A  -  ESTIMATING  THE  ACQUISITION  COST  OF 
THE  ALLIANCE  GROUND  SURVEILLANCE  (AGS)  SYSTEM 


6.0  CONCLUSIONS 

Working  as  a  coherent  team,  the  Task  Group  was  successful  in  generating  a  credible,  complete,  scientifically- 
sound  independent  cost  estimate  on  the  NATO  AGS  acquisition  program.  Specific  cost  estimates  for  the  pro¬ 
gram  are  business-sensitive  information  and  cannot  be  published.  Nevertheless,  these  conclusions  arc  offered: 

•  It  is,  in  fact,  possible  to  generate  an  ICE  on  a  major  acquisition  program  using  an  international  team  of 
dedicated  cost  analysts. 

•  It  is  difficult  but  not  impossible  to  share  cost  data  across  nations.  Non-disclosure  forms  need  to  be 
completed,  and  information  closely  held.  Further,  it  is  useful  if  not  essential  to  obtain  buy-in  early  on 
from  major  stakeholders.  The  SAS-076  team  generated  an  ICE  development  plan  and  obtained  signatures 
from  the  Chairman  of  the  Board  of  AGS  Management  Organization  (NAGSMO),  the  Director  the  AGS 
Management  Agency  (NAGSMA),  and  the  Director  the  SAS-076  study  group. 

•  Major  elements  of  cost  risk  on  the  program  arc: 

-  Software  development; 

-  International  participation; 

-  Exchange  rate; 

-  Inflation  rate;  and, 

-  MP-RTIP  radar. 

•  The  SAS-054  guidelines  on  methods  and  models  for  cost  analysis  are,  in  fact,  applicable  in  a  cost¬ 
estimating  environment.  The  SAS-076  Task  Group  followed  these  guidelines  extensively  and  recom¬ 
mends  no  major  changes. 
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Appendix  A-l:  A  GENERAL  OVERVIEW  OF  THE  ADVANCED 
COST  ESTIMATING  SYSTEMS  (ACES) 

The  parametric  hardware,  software  and  built-in  life  cycle  model  ACES  is  a  computerized  method  for  deriving 
cost  estimates  of  electronic  and  mechanical  hardware  assemblies  and  systems,  as  well  as  software  programs 
and  the  complete  life  cycle  analysis.  The  ACES  model  combines  both  hardware  and  software  estimating  in  one 
model.  The  model  is  applicable  to  all  aspects  of  hardware  and  software  acquisition  from  preliminary  design  to 
development  and  production  and  life  cycle  costing  analysis. 

The  model  estimates  the  costs  associated  with  design  engineering,  drafting,  project  management,  docu¬ 
mentation,  system  engineering,  special  tooling  and  test  equipment,  and  material,  labour  and  overheads.  It  also 
addresses  the  feasibility  of  a  project  by  costing  a  preliminary  design  phase.  Costs  to  integrate  subsystems  and 
assemblies  into  a  system,  and  to  test  the  system  for  required  operation  are  also  estimated  by  the  model. 

The  model  will  compute  and  display  the  cost  for  development  and/or  production,  as  well  as  the  software 
cost  categories,  separately  for  the  mechanical  and  electronic  parts  of  the  hardware,  and  can  be  used  to  cost 
electronic  or  mechanical  items  individually. 

The  fundamental  parametric  data  used  by  ACES  are: 

•  Quantities  of  equipment  to  be  developed,  produced,  or  otherwise  acquired. 

•  Schedules  for  preliminary  development,  full  scale  development,  production,  and  integration  and  testing, 
including  lead  times  for  set-up,  parts  procurement  (non-long  lead  items),  and  redesign. 

•  Manpower  requirements  for  engineering  and  software. 

•  Hardware  geometry  consisting  of  size  and  weights  of  electronic  and  mechanical  elements. 

•  Amount  of  new  design  required  and  complexity  of  the  development  engineering  task. 

•  Operational  environment  and  development  and  production  specification  requirements. 

•  Technology  difficulty  Index  of  the  mechanical  and  electronics  portions  of  the  hardware. 

•  Fabrication  process  to  be  used  for  production. 

•  Pertinent  escalation  rates  and  mark-ups  for  general  and  administrative  charges,  profit,  R&D,  cost  of 
money,  and  material  handling. 

•  Technological  state-of-the-art. 

•  Calculating  technology  based  INDEX  factors  from  any  known  cost. 

•  Estimating  the  individual  costs  of  multiple  lot  productions. 

•  Calculating  the  costs  for  design  integration  of  purchased  equipment. 

•  Calculating  the  costs  for  design  integration  of  furnished  equipment 

•  Calculating  technology  based  indices  of  non-homogeneous  hardware  items. 

•  Modelling  any  resource  expenditure  distribution  desired. 
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•  Calculating  field  reliability  (MTBF)  of  electronic  and  mechanical  assemblies. 

All  of  the  ACES  parameters  are  dynamically  linked  to  one  or  more  other  inputs  in  computing  model  results. 
Usually,  a  change  to  one  input  will  require  a  change  to  others  as  well.  An  example  is  production  quantity.  This 
variable  has  a  direct  effect  on  schedule  and  manufacturing  process.  These  types  of  inter-dependencies  are  what 
distinguish  a  model  from  a  calculator. 
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Appendix  A-2:  ACES  INPUT  AND  OUTPUT  PARAMETER  DEFINITIONS 

Tables  30-32  define  the  input  and  output  parameters  as  they  apply  to  they  apply  to  the  Harris  RF-9070  UHF/VHF 
antenna. 
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Table  30:  Input  definitions  for  ACES  parameters  as  they  apply  to  the  Harris  RF-9070  UHF/VHF  antenna 


Parameter 

Value 

Description 

Specification 

Mode 

22 

The  MODE  defines  the  input  required  to  describe  the 
hardware  development  and/or  production  scenario. 

MODE  22  is  used  for  an  item  that  is  mechanical 
or  structural. 

WM 

9.1 

WM  is  the  weight  of  mechanical  item  in  kilograms. 

Value  from  the  specification. 

LQTY 

1000 

Lot  quantity  indicates  the  total  size  of  a  production  lot  when 
only  a  portion  of  the  larger  production  lot  is  used  by  the 
element. 

Assumption  that  the  Antenna  is  part  of  a 
larger  lot. 

ENVIRD 

1.6 

In  general  this  variable  relates  the  cost  of  hardware 
development  to  the  requirements  of  the  environment  in  which 
the  hardware  must  operate.  It  is  a  measure  of  the  portability, 
reliability,  structure,  testing  and  documentation  required  for 
acceptable  contract  performance. 

The  recommended  value  for  mobile  military 
equipment. 

ENVIRP 

1.6 

In  general  this  variable  relates  the  cost  of  hardware 
production  to  the  requirements  of  the  environment  in  which 
the  hardware  must  operate.  It  is  a  measure  of  the  portability, 
reliability,  structure,  testing  and  documentation  required  for 
acceptable  contract  performance. 

The  recommended  value  for  mobile  military 
equipment. 

QTY 

11 

Production  Quantity  to  indicate  the  number  of  production  units, 
for  the  element. 

Equal  to  the  number  of  ground  stations. 

UCL 

0.917 

Unit  Learning  Curve 

A  typical  value  for  military  equipment. 

PROTO 

0 

Number  of  prototypes 

Assumes  the  antenna  is  a  purchased  item  and 
no  prototype  was  necessary. 

PROITE 

1 

Prototype  iterations  indicate  the  number  of  design/redesign 
cycles  needed  to  complete  the  final  prototype 

Using  a  standard  factor  for  generating  design 
integration  cost. 

NEWREPM 

0.05 

New/Repeat  mechanical  design 

Using  a  low  factor  for  generating  design 
integration  cost. 

ENGDIF 

1.4957 

Engineering  difficulty 

Using  recommended  mean  values 

INDEXM 

4.51 

INDEXM  is  a  measure  of  the  equipment  item’s  technology,  its 
produceability  (material,  machining  and  assembly  tolerances, 
machining  difficulty  and  surface  finish,  etc.). 

Using  recommended  values  from  an  index  table 
inside  the  tool  or  calibrated  values  from  other 

sources. 

INTEGM 

4.51 

Integration  difficulty  for  mechanic.  INTEGM  defines  the  level  of 
the  structural  contribution  to  the  Integration  and  test  effort  for 
this  element. 

A  Table  and  Generator  from  the  tool  is  provided 
to  suggest  appropriate  values. 

START  PD  -  NTH  PRT 

Start  and  end  date  for  the  development. 

Dates  from  the  program  office. 

START  P  -  NTH  P 

Start  and  end  date  for  the  production. 

Dates  from  the  program  office. 
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Table  31:  Output  definitions  for  ACES  parameters  as  they  apply  to  the  Harris  RF-9070  UHF/VHF  antenna 


Cost  Element 

Development 

Production  Total  Cost 

Concept  Draft,  Concept  Design, 

These  cost  elements  contain  all  costs 

These  cost  elements  contain  all  costs 

Full  Draft,  Detail  Design 

for  drafting  the  Detailed  Design  in  the 
development  phase. 

for  drafting  the  Detailed  Design  in  the 
production  phase. 

System 

This  element  encompasses  the  system 
engineering  effort  to  define  the  system, 
and  integrated  planning  and  control  of 
the  technical  program  efforts  of  design 
engineering,  logistic  engineering, 
production  engineering  and  integrated 
test  planning. 

Same  as  the  development  but  limited  to 
production  needs. 

Project  Management 

Project  Management  includes 
administrative  and  technical 
management  and  other  program- 
oriented  personnel.  These  costs  include 
all  efforts  to  assure  the  proper  and  cost- 
effective  performance  of  the  contract 
between  the  customer,  the  company 
and  its  subcontractors. 

This  cost  element  is  similar  to  the 
development  effort  but  limited  to 
monitoring  and  controlling  production 
activities. 

Documentation 

Prototype 

This  cost  element  contains  all  costs 
associated  with  deliverable 
documentation. 

These  costs  include  all  labour  and 
material  required  to  fabricate,  assemble 
and  test  the  prototypes. 

This  cost  element  is  similar  to  the 
development  effort  but  limited  to 
production  items. 

Tooling 

This  cost  element  contains  all  costs 
associated  with  the  design,  fabrication, 
purchase  and  calibration  of  all  special 
tools,  fixtures  and  test  equipment 
required  to  handle,  build  and  test 
prototype  components. 

Same  as  the  development  category  but 
includes  production  related  tooling. 
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Table  32:  Output  definitions  (Cont’d)  for  ACES  parameters  as  they  apply  to  the  Harris  RF-9070  UHF/VHF  antenna 


Cost  Element 

Development 

Production 

Total  Cost 

Production 

This  cost  element  includes  all 
labour  costs  involved  in 
the  recurring  manufacturing  efforts 
to  fabricate,  assemble.  It  also 
includes  all  material  that  is 
contained  in  the  completed 
article. 

Integration  &  Testing 

This  cost  element  includes  all 
costs  associated  with  the  test 
and  integration  for  this 
element  into  the  system. 

This  cost  element  includes  all 
costs  associated  with  the  test 
and  integration  for  this  element 
in  the  system. 

T-l  Cost 

Cost  for  the  first  unit. 

Cost  for  the  first  unit. 

HKl/STk 

(Unit  Production  Cost  (UPC)) 

Unit  production  cost  (The  unit 
production  cost  for  the 
specified  production  quantity. 

It  includes  production  labour, 
material  and  overhead.) 

Unit  production  cost  (The  unit 
production  cost  for  the 
specified  production  quantity. 

It  includes  production  labour, 
material  and  overhead.) 

HK2/STk  (Amortized  UPC/Quantity) 

HK2+Sw  (Amortized  UPC  +  Software) 

Amortized  Unit  Cost  (The 
Amortized  Unit  Cost  is 

calculated  as  Production  Total 
Cost  divided  by  Quantity) 

Amortized  Unit  Cost  plus 
Software  costs 

Amortized  Unit  Cost  (The 
Amortized  Unit  Cost  is 

calculated  as  Production  Total 

Cost  divided  by  Quantity) 

Amortized  Unit  Cost  plus 

Software  costs 

Ges/Stk  (Total  Sum/Quantity) 

Total  costs  divided  by  the  quantity 
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ANNEX  B1  -  ESTIMATING  THE  ACQUISITION 
COST  OF  THE  ROYAL  NETHERLANDS  NAVY 
LANDING  PLATFORM  DOCK  SHIPS:  AN  EX  POST  ANALYSIS 


1.0  INTRODUCTION 
1.1  Background 

This  paper  documents  the  SAS-076  Task  Group  efforts  in  generating  independent  estimates  of  the  development 
and  construction  costs  of  the  Royal  Netherlands  Navy  landing  platform  dock  ships  in  a  blind,  ex  post  exercise — 
only  after  the  independent  cost  estimates  were  completed  did  the  Task  Group  obtain  information  on  the  actual 
cost  of  landing  platform  dock  ships.  The  differences  between  the  actual  and  estimated  costs  were  then  analyzed. 

The  Rotterdam  class  is  a  Landing  Platform  Dock  (LPD)  or  amphibious  warfare  ship  of  the  Royal  Nether¬ 
lands  Navy.  The  lead  ship.  Her  Netherlands  Majesty’s  Ship  (HMS)  Rotterdam,  pennant  number  L800,  was 
launched  in  1997.  The  second  ship  of  the  class,  HMS  Johan  de  Witt  (L801),  was  launched  in  2007.  The  ships 
have  a  large  helicopter  deck  and  a  well  dock  for  large  landing  craft.  The  class  was  a  joint  design  between 
the  Netherlands  and  Spain1.  HMS  Rotterdam  is  pictured  in  Figure  1.  Although  considered  the  sister  ship  to 
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Figure  1 :  HMS  Rotterdam  L800  Landing  Platform  Dock  Ship 

HMS  Rotterdam,  HMS  Johan  de  Witt  has  significant  technical  differences  including  longer  length,  larger  dis¬ 
placement,  podded  propulsion  (as  oppose  to  shaft  propulsion),  increased  lift  capacity  (e.g.,  fuel,  vehicles,  etc.), 
higher  superstructure  (by  one  deck),  and  larger  crew  capacity.  For  this  reason  HMS  Johan  de  Witt’s  rank  in 
class  is  set  to  one. 

Following  a  key  recommendation  of  SAS-054,  this  report  documents  two  independent  methods  that  were 
used  in  generating  a  cost  estimate  for  HMS  Rotterdam  and  Johan  de  Witt.  The  first  method,  classified  as 
a  parametric  approach,  employs  Quinlan’s  M5  model  tree  algorithm  [1],  a  system  that  combines  features  of 
decision  trees  with  linear  regression  models.  The  second  method  is  an  analogy  approach  based  on  hierarchical 
cluster  analysis.  It  considers  multiple  analogous  systems  and  is  void  of  subjectivity  that  is  typically  inherent 

'The  two  ships  of  the  Spanish  belonging  to  the  Galicia  class  are  the  Galicia  (commissioned  in  1998)  and  the  Castilla  (2001). 
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in  analogy  approaches.  Both  data  mining  approaches  are  data  intensive — the  SAS-076  Task  Group  compiled  a 
database  of  57  ships  in  16  classes  from  6  nations.  136  descriptive,  technical,  and  cost  attributes  were  gathered 
for  each  of  the  ships. 

1.2  Objective 

The  objective  of  this  report  is  to  present  two  independent  methods  for  generating  estimates  for  the  development 
and  production  cost  of  HMS  Rotterdam  and  Johan  de  Witt  LPDs.  In  an  ex  post  exercise,  the  actual  costs 
are  revealed  and  compared  to  the  estimates.  The  report  presents  a  novel  application  of  known  data  mining 
algorithms  to  the  problem  of  ship  cost  estimation. 

1.3  Scope 

This  report  focuses  on  two  data  mining  approaches:  M5  model  trees  for  parametric  cost  estimation,  and  hierar¬ 
chical  cluster  analysis  for  costing  by  analogy.  Limitations  and  assumptions  specific  to  the  parametric  method 
are  presented  in  Sections  3.0  and  4.0. 

1.4  Outline 

Section  2.0  details  the  multinational  data  that  was  gathered  (Annex  ??  contains  data  tables  continuing  from 
Section  2.0).  The  parametric  cost  estimation  method  is  developed  in  Section  3.0  and  applied  to  the  data  set. 
The  analogy  cost  estimation  method  is  developed  in  Section  4.0  and  applied  to  the  data  set.  The  results  for  the 
predicted  cost  of  HMS  Rotterdam  and  Johan  de  Witt  LPDs  are  presented  in  Sections  5.0  and  in  particular  are 
compared  to  the  actual  costs.  Section  6.0  compiles  lessons  identified,  conclusions  are  presented  in  Section  7.0, 
and  Annex  ??  provides  the  algorithmic  details  of  the  approach. 

2.0  DATA 

The  SAS-076  Task  Group  compiled  a  database  of  57  ships  in  16  classes  from  6  nations.  The  sources  of  data  were 
culled  from  SAS-076  participants  and  publicly-available  sources  such  as  Jane’s  Fighting  Warships2,  Federation 
of  American  Scientists3,  Navy  Matters4,  Forecast  International5,  U.S.  Naval  Institute  sources  (e.g.,  [2]),  and 
Wikipedia:  The  Free  Encyclopedia6.  The  ships  included  cover  a  span  of  years  (commissioned)  from  1954  to 
2010.  Table  1  lists  the  ships  in  the  data  set.  The  first  three  columns  indicate  the  name,  pennant  number,  and 
type  of  ship.  The  subsequent  three  columns  indicate  the  rank  (in  class),  year  of  commission,  and  nationality. 

The  ships  included  cover  a  span  of  years  (commissioned)  from  1954  to  2010.  Table  1  lists  the  ships  in  the 
data  set.  The  first  three  columns  indicate  the  name,  pennant  number,  and  type  of  ship.  The  subsequent  three 
columns  indicate  the  rank  (in  class),  year  of  commission,  and  nationality.  The  data  set  includes  seven  ship 
categories: 

•  Amphibious  Assault  Ship  (AAS); 

•  Auxiliary  Oiler  Replenishment  (AOR); 

2Jane’s  Fighting  Warships:  http://jfs.janes.com 

3Federation  of  American  Scientists:  http://www.fas.org/ 

4Navy  Matters:  http://www.navy-matters.beedall.com 

5  Forecast  International:  http://www.forecastinternational.com 

6Wikipedia:  The  Free  Encyclopedia:  http://www.wikipedia.com 
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•  Landing  Platform  Dock  (LPD); 

•  Landing  Platform  Helicopter  (LPH); 

•  Landing  Ship  Dock  (LSD);  and, 

•  Icebreaker. 

Forty  of  the  ships  are  from  the  U.S.,  seven  from  the  United  Kingdom,  four  from  France,  three  from  Sweden, 
two  from  Canada,  and  one  from  Norway. 

The  SAS-076  ship  data  set  contains  military  and  civilian  auxiliary  (coast  guard  or  similar)  vessels  that 
were  judged  by  the  SAS-076  Task  Group  to  be  analogous  to  HMS  Rotterdam  LPD.  The  selection  of  ships  for 
inclusion  was  primarily  driven  by  the  accessibility  of  costing  information,  which  was  limited  to  the  information 
gathered  from  national  sources  by  the  SAS-076  Task  Group  representatives7.  Representatives  of  the  SAS- 
076  Task  Group  were  solicited  to  provide  technical  and  cost  information  for  their  nation’s  ships  most  closely 
resembling  the  role  or  size  of  a  LPD.  The  decision  as  to  which  ships  are  similar  or  analogous  is  subjective. 

The  novel  parametric  approach  to  cost  estimation  using  data  mining  (to  be  elaborated  in  Section  3.0)  allows 
for,  or  can  excel  on,  a  greater  variability  in  the  input  data  set — variability  that  could  be  questioned  when  using 
traditional  parametric  approaches.  The  fidelity  of  parametric  approaches  also  depends  on  the  size  of  the  input 
data  set.  In  this  case,  the  availability  of  ship  cost  data  limited  the  size  of  the  SAS-076  ship  data  set.  For  these 
reasons,  the  SAS-076  Task  Group  decided  to  include  ships  such  as  Sweden’s  Carlskrona  LPD  and  Atle  and 
Oden  icebreakers — leading  to  the  potentially  useful  data  combination  of  ships  of  the  “right  purpose,  wrong 
size”  and  “wrong  puipose,  right  size”  [3], 

The  SAS-076  Task  Group  also  decided  to  omit  the  U.S.  San  Antonio  class  LPDs  due  to  its  well-documented 
high  cost  of  ship  construction:  as  of  July  2007,  the  U.S.  Navy  had  invested  over  1.75  billion  U.S  dollars 
constructing  the  San  Antonio  LPD  17,  over  800  million  dollars  over  budget  [4]  (including  a  mid-construction 
transfer  from  Avondale  shipyard  in  New  Orleans  to  the  Northrop  Grumman  shipyard  in  Mississippi  due  to  poor 
performance  by  the  former).  According  to  the  U.S.  Government  Accountability  Office  [5]:  “the  U.S.  Navy’s 
reliance  on  an  immature  design  tool  led  to  problems  that  affected  all  aspects  of  the  lead  ship’s  design.  Without 
a  stable  design,  work  was  often  delayed  from  early  in  the  building  cycle  to  later,  during  integration  of  the  hull”. 
The  LPD- 17  cost  was  a  clear  outlier  in  comparison  to  the  costs  of  the  other  ships.  It  was  deemed  that  its  cost 
was  not  accurately  reflective  of  its  technical  attributes. 

The  selection  of  technical  specifications  included  was  also  driven  by  the  availability  of  public  information. 


7For  example,  detailed  technical  information  was  available  for  the  Italian  San  Giorgio  class  and  the  Spanish  Galicia  class  LPDs,  but 
cost  information  was  unobtainable 
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Table  1 :  Description  of  analogous  ships 


Name 

Number 

Type 

Rank 

Commissioned 

Country 

Thomaston 

LSD  28 

LSD 

1 

1954 

United  States 

Plymouth  Rock 

LSD  29 

LSD 

2 

1954 

United  States 

Fort  Snelling 

LSD  30 

LSD 

3 

1955 

United  States 

Point  Defiance 

LSD  31 

LSD 

4 

1955 

United  States 

Spiegel  Grove 

LSD  32 

LSD 

5 

1956 

United  States 

Alamo 

LSD  33 

LSD 

6 

1956 

United  States 

Hermitage 

LSD  34 

LSD 

7 

1956 

United  States 

Monticello 

LSD  35 

LSD 

8 

1957 

United  States 

Anchorage 

LSD  36 

LSD 

1 

1969 

United  States 

Portland 

LSD  37 

LSD 

2 

1970 

United  States 

Pensacola 

LSD  38 

LSD 

3 

1971 

United  States 

Mount  Vernon 

LSD  39 

LSD 

4 

1972 

United  States 

Fort  Fisher 

LSD  40 

LSD 

5 

1972 

United  States 

Whidbey  Island 

LSD  41 

LSD 

1 

1985 

United  States 

Germantown 

LSD  42 

LSD 

2 

1986 

United  States 

Fort  McHenry 

LSD  43 

LSD 

3 

1987 

United  States 

Gunston  Hall 

LSD  44 

LSD 

4 

1989 

United  States 

Comstock 

LSD  45 

LSD 

5 

1990 

United  States 

Tortuga 

LSD  46 

LSD 

6 

1990 

United  States 

Rushmore 

LSD  47 

LSD 

7 

1991 

United  States 

Ashland 

LSD  48 

LSD 

8 

1992 

United  States 

Harpers  Ferry 

LSD  49 

LSD 

1 

1995 

United  States 

Carter  Hall 

LSD  50 

LSD 

2 

1995 

United  States 

Oak  Hill 

LSD  51 

LSD 

3 

1996 

United  States 

Pearl  Harbour 

LSD  52 

LSD 

4 

1998 

United  States 

Raleigh 

LPD  1 

LPD 

1 

1962 

United  States 

Vancouver 

LPD  2 

LPD 

2 

1963 

United  States 

La  Salle 

LPD  3 

LPD 

3 

1964 

United  States 

Austin 

LPD  4 

LPD 

1 

1965 

United  States 

Ogden 

LPD  5 

LPD 

2 

1965 

United  States 

Duluth 

LPD  6 

LPD 

3 

1965 

United  States 

Cleveland 

LPD  7 

LPD 

4 

1967 

United  States 

Dubuque 

LPD  8 

LPD 

5 

1967 

United  States 

Denver 

LPD  9 

LPD 

6 

1968 

United  States 

Juneau 

LPD  10 

LPD 

7 

1969 

United  States 

Coronado 

LPD  1 1 

LPD 

8 

1970 

United  States 

Shreveport 

LPD  12 

LPD 

9 

1970 

United  States 

Nashville 

LPD  13 

LPD 

10 

1970 

United  States 

Trenton 

LPD  14 

LPD 

11 

1971 

United  States 

Ponce 

LPD  15 

LPD 

12 

1971 

United  States 

Svalbard 

W303 

Icebreaker 

1 

2001 

Norway 

Carlskrona 

M04 

LPD 

1 

1982 

Sweden 

Atle 

— 

Icebreaker 

1 

1985 

Sweden 

Oden 

— 

Icebreaker 

1 

1989 

Sweden 

Protecteur 

AOR  509 

AOR 

1 

1969 

Canada 

Preserver 

AOR  510 

AOR 

2 

1970 

Canada 

Albion 

L14 

LPD 

1 

2003 

United  Kingdom 

Bulwark 

L15 

LPD 

2 

2005 

United  Kingdom 

Largs  Bay 

L3006 

LSD 

1 

2006 

United  Kingdom 

Lyme  Bay 

L3007 

LSD 

2 

2007 

United  Kingdom 

Mounts  Bay 

L3008 

LSD 

3 

2006 

United  Kingdom 

Cardigan  Bay 

L3009 

LSD 

4 

2006 

United  Kingdom 

Ocean 

L12 

LPH 

1 

1998 

United  Kingdom 

Siroco 

L9012 

LSD 

2 

1998 

France 

Mistral 

L9013 

AAS 

1 

2006 

France 

Tonnerre 

L9014 

AAS 

2 

2007 

France 

Dixmude  (BPC3) 

L9015 

AAS 

3 

2010 

France 
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2.1  Technical  Data 

Descriptive,  technical,  and  cost  data  was  gathered  for  each  of  the  ships  in  the  SAS-076  data  set.  The  list  of 
these  ship  attributes  are  broken  down  into  the  categories  as  per  Table  2.  Tables  15  and  16  in  Annex  ??  detail  the 
technical  attributes  (categories  III  to  XIV  of  Table  2)  of  the  17  ship  classes.  Table  3  details  the  attributes  as  well 
as  the  input  for  HMS  Rotterdam  and  Johan  de  Witt  ships.  Attribute  units  are  expressed  by  either  nominal  values 
(e.g.,  “fixed  pitch”  (fp),  “controlled  pitch”  (cp),  “yes”  (Y),  “no”  (N)),  or  by  numerical  units  such  as  meters  (m), 
milimeters  (mm),  megawatts  (MW),  knots  (kts),  hours  (hrs),  nautical  miles  (nmi),  etc.  Unknown  or  missing 
data  is  denoted  by  “?”  entries.  For  conciseness,  undefined  technical  acronyms  appearing  in  Tables  2  or  3  are 
listed  at  the  end  of  the  document. 


Table  2:  Categories  of  ship  data 


Category 

Number  of  Attributes 

I 

DESCRIPTION 

6 

II 

CONSTRUCTION 

8 

III 

DIMENSIONS 

5 

IV 

PERFORMANCE 

8 

V 

PROPULSION 

9 

VI 

ELECTRICAL  POWER  GENERATION 

3 

VII 

LIFT  CAPACITY 

35 

VIII 

FLIGHT  DECK 

19 

IX 

ARMAMENT 

13 

X 

COUNTERMEASURES 

5 

XI 

RADARS  /  TACAN  /  IFF  /  SONARS 

13 

XII 

COMBAT  DATA  SYSTEMS 

1 

XIII 

WEAPONS  CONTROL  SYSTEMS 

1 

XIV 

OTHER  CAPABILITIES 

7 

XV 

COST  DATA 

3 

Total: 

136 

Table  3:  Complete  list  of  ship  data  for  the  Rotterdam  and  Johan  de  Witt  LPDs 


Data  Category  &  Element 

Rotterdam  LPD 

Johan  de  Witt  LPD 

I.  DESCRIPTION 

a.  Name 

Rotterdam 

Johan  de  Witt 

b.  Nation 

Netherlands 

Netherlands 

c.  Number 

L800 

L801 

d.  Type 

Landing  Platform  Dock 

Landing  Platform  Dock 

e.  Rank  in  class 

I 

1 

f.  Vessel  type  (Military  /  Civilian) 

Military 

Military 

II.  CONSTRUCTION 

a.  Laid  down 

1/25/1996 

6/18/2003 

b.  Launched 

2/22/1997 

5/13/2006 

c.  Commissioned 

4/18/1998 

1 1/30/2007 

d.  Shipyard 

Royal  Schelde 

Royal  Schelde 

e.  City 

Vlissingen 

Vlissingen 

f.  Country 

Netherlands 

Netherlands 

continued  on  next  page 
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continued  from  previous  page 


Data  Category  &  Element 

Rotterdam  LPD 

Johan  de  Witt  LPD 

g.  Continent 

Europe 

Europe 

h.  Built  to  civilian  classification  society  standards? 

Y 

Y 

III.  DIMENSIONS 

a.  Length  (m) 

162.2 

176.35 

b.  Beam  (m) 

25 

25 

c.  Draught  (m) 

5.9 

5.9 

d.  Displacement  (tonnes)  Light  load 

8410 

11560 

e.  Displacement  (tonnes)  Full  load 

12750 

16680 

IV.  PERFORMANCE 

a.  Top  speed  (kts) 

19 

19 

i.  Range:  Total  distance  (nmi) 

6000 

10000 

ii.  Range:  Economical  speed  (kts) 

12 

12 

iii.  Range:  Sailing  time  (hours) 

500 

833 

b.  Endurance  (days) 

42 

42 

c.  Crew:  complement 

113 

146 

i.  Officers 

13 

17 

ii.  Non-officers 

100 

129 

V.  PROPULSION 

a.  Propulsion  technology 

Electric 

Electric 

b.  Propeller  shafts 

2 

0 

i.  Shaft  propulsion  power  (MW) 

12 

0 

ii.  Propeller  type 

fixed  pitch 

N/A 

c.  Propulsion  pods 

0 

2 

i.  Total  podded  propulsion  power  (MW) 

0 

11 

d.  Net  propulsion  power  (MW) 

12 

11 

e.  Bow  Thrusters 

1 

2 

i.  Total  thruster  power  (MW) 

1.15 

1.8 

VI.  ELECTRICAL  POWER  GENERATION 

a.  Generators 

4 

4 

i.  Total  power  generation  capacity  (MW) 

14.6 

14.4 

ii.  Generator  technology 

Diesel 

Diesel 

VII.  LIFT  CAPACITY 

a.  Vehicle  fuel  (litres) 

9000 

14500 

b.  Aviation  fuel  (litres) 

284400 

306600 

c.  Fresh  water  (litres) 

263100 

329900 

d.  Bulk  cargo  space  (/w3) 

3680 

4170 

e.  Vehicle  space  ( m 2) 

720 

1770 

f.  Well  deck  (Y/N) 

Y 

Y 

i.  Length  (m) 

50 

35 

ii.  Width  (m) 

14 

15 

iii.  Capacity  (m2) 

700 

525 

iv.  LCAC 

0 

0 

v.  LCM  6 

? 

? 

vi.  LCM  8 

4 

4 

vii.  LCU 

4 

4 

viii.  LVT 

? 

? 

ix.  LCVP 

6 

6 

x.  LCPL 

? 

? 

xi.  EFV 

? 

? 

g.  Cargo/ Aircraft  Elevator/Lifts 

0 

0 

i.  Capacity  <  5  tonnes 

0 

0 

ii.  5  <  capacity  <10  tonnes 

0 

0 

iii.  10  <  capacity  <  15  tonnes 

0 

0 

iv.  Capacity  >15  tonnes 

0 

0 

h.  Cranes 

1 

1 

i.  Capacity  <  5  tonnes 

0 

0 

ii.  5  <  capacity  <10  tonnes 

0 

0 

iii.  10  <  capacity  <  15  tonnes 

0 

0 

iv.  1 5  <  capacity  <  20  tonnes 

0 

0 

v.  20  <  capacity  <  25  tonnes 

1 

1 

vi.  25  <  capacity  <  30  tonnes 

0 

0 

vii.  30  <  capacity  <  40  tonnes 

0 

0 

viii.  40  <  capacity  <  50  tonnes 

0 

0 

ix.  50  <  capacity  <  60  tonnes 

0 

0 

x.  Capacity  >  60  tonnes 

0 

0 

i.  Berthing  (troop  capacity):  Baseline 

611 

555 

j.  Berthing  (troop  capacity):  Surge 

100 

100 

VIII.  FLIGHT  DECK 

a.  Equipped  with  flight  deck?  (Y/N) 

Y 

Y 

b.  Flight  deck  length  (m) 

56 

58 

c.  Flight  deck  width  (m) 

25 

25 

d.  Flight  deck  area  (m2) 

1400 

1450 

e.  Helicopter  landing  spots  (maximum  number) 

2 

2 

continued  on  next  page 
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Data  Category  &  Element 

Rotterdam  LPD 

Johan  de  Witt  LPD 

i.  Merlin  /  Sea  King 

2 

2 

ii.  NH  90  /  Lynx  /  Puma  /  Cougar 

2 

2 

iii.  CH-46E  Sea  Knight 

? 

? 

iv.  CH-53  Sea  Stallion 

? 

? 

v.  MV-22  Osprey 

? 

? 

f.  Chinook  capable  (Yes/No) 

N 

Y 

g.  Equipped  with  hangar?  (Y/N) 

Y 

Y 

h.  Hangar  size  (m2) 

475 

560 

i.  Helicopters  supported  (largest  total  number) 

6 

6 

i.  Merlin  /  Sea  King 

4 

4 

ii.  NH  90  /  Lynx  /  Puma  /  Cougar 

6 

6 

iii.  CH-46E  Sea  Knight 

7 

? 

iv.  CH-53  Sea  Stallion 

? 

? 

v.  MV-22  Osprey 

? 

? 

IX.  ARMAMENT 

a.  Guns  (calibre  >  75mm) 

0 

0 

b.  Guns  (50mm  <  calibre  <  75mm) 

0 

0 

c.  Guns  (30mm  <  calibre  <  50mm) 

0 

0 

d.  Guns  (20mm  <  calibre  <  30mm) 

0 

0 

e.  30mm  CIWS  emplacements  (Goalkeeper) 

2 

2 

f.  20mm  CIWS  emplacements  (Phalanx) 

0 

0 

g.  Machine  guns  (12.7mm) 

8 

4 

h.  Machine  guns  (7.62mm) 

0 

0 

i.  SSM  launchers 

0 

0 

j.  SAM  launchers 

0 

0 

k.  Number  of  torpedoes  carried 

0 

0 

1.  Torpedo  tubes 

0 

0 

m.  Torpedo  launchers 

0 

0 

X.  COUNTERMEASURES 

a.  Chaff  launchers 

4 

4 

b.  Torpedo  decoys 

1 

1 

c.  Other  systems 

0 

0 

d.  Number  of  ESM  systems 

1 

1 

e.  Number  of  ECM  systems 

1 

1 

XI.  RADARS  /  TACAN  /  IFF  /  SONARS 

a.  Total  radar  systems  mounted 

4 

5 

i.  A-band 

0 

0 

ii.  B-band 

0 

0 

iii.  C-band 

0 

0 

iv.  D-band 

0 

0 

v.  E-band 

1 

2 

vi.  F-band 

1 

2 

vii.  G-band 

0 

1 

viii.  H-band 

0 

0 

ix.  I-band 

3 

3 

x.  J-band 

0 

0 

b.  Number  of  TACAN/IFF  systems  mounted 

1 

1 

c.  Number  of  distinct  sonar  systems  mounted 

0 

0 

XII.  COMBAT  DATA  SYSTEMS 

a.  Number  of  distinct  systems 

2 

2 

XIII.  WEAPONS  CONTROL  SYSTEMS 

a.  Number  of  distinct  systems 

1 

1 

XIV.  OTHER  CAPABILITIES 

a.  Equipped  with  hospital?  (Y/N) 

Y 

Y 

i.  Number  of  beds 

10 

7 

ii.  Operating  rooms 

1 

1 

iii.  X-Ray  facility  (Y/N) 

Y 

Y 

b.  Dental  capability  (Y/N) 

Y 

Y 

c.  Command/Control  facility  (Y/N) 

Y 

Y 

d.  NBCD  Facilities  (Y/N) 

Y 

Y 

XV.  COST  DATA 

a.  Base  year 

? 

? 

b.  Currency 

EUR 

EUR 

c.  Development  and  Production  Cost 

? 

? 

2.2  Cost  Data 

The  SAS-076  ship  data  set  includes  ship  costs  expressed  in  various  currencies:  Great  Britain  pound  sterling 
(GBP),  U.S.  dollars  (USD),  Canadian  dollars  (CAD),  Norwegian  krone  (NOK),  Swedish  kronor  (SEK),  and 
Euros  (EUR).  Costs  are  also  expressed  in  various  then-years8,  ranging  from  1952  to  2009.  The  SAS-054 
recommendations  for  cost  normalization  of  multinational  programs  are  summarized  as  follows: 


8  Amounts  that  include  the  effects  of  inflation  or  escalation,  and/or  reflect  the  price  levels  expected  to  prevail  during  the  year  at  issue. 
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1.  apply  escalation  formulae  using  indices  from  supplier  nations  to  model  changes  due  to  national  inflation 
prior  to  conversion  to  other  currencies; 

2.  each  nation  should  apply  its  own  cost  inflation  model  based  on  applicable  data  and  nation  specific  eco¬ 
nomic  advice  that  is  pertinent  to  the  particular  system;  and, 

3.  use  exchange  rates  between  the  national  currencies  as  the  currency  conversion  mechanism. 

The  SAS-076  Task  Group  followed  the  SAS-054  guidelines  to  normalize  the  ship  costs  to  a  common  cur¬ 
rency  and  then-year.  Various  inflation  rates  were  research,  and  in  particular  the  Task  Group  was  guided  by  U.S. 
Naval  Sea  Systems  Command  (NAVSEA)  representatives  who  were  not  aware  of  any  “publicly  available  total 
ship  cost  index  that  goes  back  to  the  1950s”  [6].  The  U.S.  Bureau  of  Labour  Statistics  [7]  has  a  shipbuild¬ 
ing  labour  index,  and  there  are  various  materiel  indices  that  are  applicable  to  shipbuilding,  but  the  high-level 
data  gathered  in  the  SAS-076  data  set  is  not  detailed  enough  to  compute  a  weighted  overall  composite  index. 
Instead,  inflation  rate  sources  were  obtained  as  follows: 

•  The  SAS-076  U.S.  representative  from  the  Office  of  the  Assistant  Secretary  of  the  Navy  obtained  the 
Historical  Shipbuilding  and  Conversion,  Navy  (SCN)  Total  Obligational  Authority  Index  used  within  the 
U.S.  NAVSEA9. 

•  To  inflate  the  cost  figures  of  ships  built  within  countries  of  the  European  Union,  the  EuroStat  Labour 
Inflation  rates  [8]  specific  to  each  country  were  applied. 

•  To  inflate  the  cost  of  Canadian-built  ships,  the  Defence  Research  &  Development  Canada  Centre  for 
Operational  Research  &  Analysis  Defence  Economics  Team  recommended  the  Industrial  Production 
Index  (IPI)  [9]. 

•  Upon  consultation  with  FOI — Swedish  Defence  Research  Agency,  the  IPI  was  also  used  to  inflate  the 
cost  of  the  Scandinavian-built  ships  for  which  pre-dated  the  available  EuroStat  indices. 

Respecting  the  anonymity  request  of  some  of  the  nations,  the  ship  costs  cannot  be  made  explicit.  For  the 
same  reason,  the  common  currency  and  then-year  are  not  disclosed — all  subsequent  cost  figures  are  presented 
using  a  fictitious  notional  common  currency  (abbreviated  NCC).  Figure  2  illustrates  the  histogram  of  the  SAS- 
076  data  set  costs  normalized  to  NCC. 

The  SAS-076  Task  Group  log-transformed  the  cost  of  each  ship  as  input  to  the  estimation  model.  The 
models,  to  be  presented  in  detail  in  Subsections  3.0  and  4.0,  output  a  single  predicted  ship  cost  in  log-space. 
This  prediction  is  considered  to  center  a  normal  distribution  whose  standard  deviation  is  the  standard  deviation 
of  the  model  (also  in  log  space)  in  estimating  the  costs  of  the  known  ships.  As  a  result  of  the  initial  log- 
transformation  of  the  costs,  the  uncertainty  in  the  prediction  of  a  ship’s  cost  is  presented  by  a  log-normal 
distribution.  The  log-normal  distribution  is  a  probability  distribution  of  a  random  variable  whose  logarithm  is 
normally  distributed. 

The  logarithmic  transformation  is  commonly  used  for  positive  data;  the  log-normal  distribution  domain 
of  zero  to  infinity  is  more  suitable  for  modelling  ship  costs  than  a  normal  distribution  which  includes  the 
negative  domain.  Log-transformation  is  also  commonly  applied  when  the  data  ranges  over  several  orders  of 
magnitude — the  SAS-076  data  set  cost  range  from  50  to  700  million  NCC.  The  majority  of  total  cost  estimates 
for  weapon-system  acquisition  programs  modeled  by  the  United  States’  Deputy  Assistant  Secretary  of  the  Navy 

9Provided  under  the  condition  that  the  indices  would  not  be  published. 
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Figure  2:  Histogram  of  normalized  costs  (millions  NCC). 


(Cost  &  Economics)  are  log-normally  distributed  and  often  skewed  right  [10].  The  log-normal  distribution  is 
also  justifiable  choice  when  a  variable  is  known  to  be  the  multiplicative  product  of  many  independent  random 
variables  each  of  which  is  positive  (as  is  usually  the  case  in  shipbuilding).  Beaulieu  et  al.  [11]  showed  that 
the  distribution  of  the  sum  of  log-normal  random  variables  can  be  approximated  with  a  single  log-normal 
distribution. 

3.0  PARAMETRIC  COST  ESTIMATION 

Parametric  approaches  to  cost  estimation  use  regression  or  other  statistical  methods  to  develop  Cost  Estimat¬ 
ing  Relationships  (CERs).  A  CER  is  an  equation  used  to  estimate  a  given  cost  element  using  an  established 
relationship  with  one  or  more  independent  variables. 

There  are  several  advantages  of  a  parametric  approach  to  cost  estimation,  including: 

•  it  can  capture  major  portions  of  an  estimate  quickly  and  with  limited  information  (as  is  often  the  case  in 
early  phases  of  the  procurement  cycle); 

•  the  CER  is  objective — it  is  based  on  consistent  and  quantitative  inputs; 

•  once  the  CER  is  established  it  is  easy  to  perform  sensitivity  analyses  (determine  the  change  in  cost  subject 
to  changes  in  the  independent  input  variables);  and, 

•  CERs  established  using  regression  analyses  include  standard  tests  of  validity,  including  a  coefficient  of 
correlation  indicating  the  strength  of  association  between  the  independent  variables  and  the  dependent 
variable  in  the  CER. 
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A  critical  consideration  in  parametric  cost  estimating  is  the  similarity  of  the  systems  in  the  underlying  database, 
both  to  each  other  and  to  the  system  which  is  being  estimated.  A  major  disadvantage  of  typical  parametric 
approaches  is  that  they  may  not  provide  low  level  visibility  (cost  breakdown)  and  changes  in  sub-systems  are 
not  reflected  in  the  estimate  if  they  are  not  quantified  via  an  independent  variable. 

Traditional  ship  building  CERs  arc  often  mathematically  simple  (e.g.  a  simple  ratio)  or  involve  linear  re¬ 
gression  analysis  (of  historical  systems  or  subsystems)  on  a  single  parameter  (weight,  length,  density,  etc.). 
However,  Miroyannis  [12]  noted  that  this  is  often  insufficient  and  that  other  cost  driving  factors  must  be  incor¬ 
porated  to  develop  estimates  of  sufficient  quality  at  the  preliminary  design  phase.  Furthermore,  the  relationship 
between  the  parameter(s)  and  cost  may  not  be  best  expressed  in  linear  form.  While  the  field  of  regression 
analysis  offers  a  multitude  of  alternative  approaches  (the  reader  is  referred  to  [13]  for  further  information), 
linear  regression  is  the  most  popular  and  easiest  to  understand. 

The  next  section  describes  a  novel  parametric  approach  for  ship  cost  estimation  that  incorporates  a  multi¬ 
tude  of  cost  driving  factors,  while  remaining  a  “top  down”  approach  applicable  in  early  design  phases  of  the 
procurement  cycle.  It  combines  features  of  decision  trees  with  linear  regression  models  to  both  classify  similar 
ships  (based  on  attributes)  and  build  piece-wise  multivariate  linear  regression  models. 


3.1  The  M5  Model  Tree  System 

In  1992  Quinlan  [1]  pioneered  the  M5  system  for  learning  models  that  predict  numeric  values.  The  M5  system 
combines  features  of  decision  trees  with  linear  regression  models.  M5  model  trees  are  similar  to  regression 
trees;  a  decision  tree  induction  algorithm  is  used  to  build  an  initial  tree,  recursively  splitting  the  data  set  based 
on  the  value  of  a  chosen  splitting  attribute.  The  splitting  attribute  is  selected  to  minimize  the  prediction  error 
down  each  branch.  Whereas  the  nodes  of  a  regression  tree  each  contain  a  constant  value  (prediction),  each 
model  tree  node  is  a  multivariate  linear  regression  model.  The  attributes  defining  these  regression  models  are 
the  attributes  that  are  involved  in  the  tree’s  branching  decisions. 

Figure  3  depicts  a  simple  example  of  a  M5  model  tree.  The  sub-figure  on  the  left  shows  a  two-dimensional 
space  of  independent  variables  x\  and  X2-  The  M5  model  tree  algorithm  splits  up  the  space  into  regions  cor¬ 
responding  to  decisions  in  the  tree  shown  on  the  right.  Finear  regression  models  are  fitted  to  the  data  in  each 
region.  To  predict  a  value  for  a  new  instance,  the  M5  model  tree  is  followed  down  to  a  leaf  using  the  instance’s 


Figure  3:  Example  M5  model  tree. 
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attribute  values  to  make  routing  decisions  at  each  node.  The  leaf  contains  a  linear  regression  model  based  on  a 
subset  of  the  attributes,  and  this  is  evaluated  for  the  new  instance  to  output  a  predicted  value. 

The  following  paragraph  synthesizes  the  M5  model  tree  algorithm.  Full  details  are  exposed  in  Annex  ??. 
The  M5  model  tree  algorithm  has  four  parts  to  it.  In  the  first  part,  a  decision  tree  is  constructed  using  the  same 
procedure  as  for  regression  trees.  The  tree  is  constructed  recursively  by  splitting  the  training  set  per  attribute 
value  chosen  to  minimize  error  in  estimation.  In  the  second  part,  the  algorithm  constructs  multivariate  linear 
models  at  each  node  of  the  model  tree  using  only  the  attributes  that  are  referenced  by  tests  somewhere  in  the 
subtree  of  this  node.  These  linear  models  are  further  simplified  by  eliminating  parameters  to  minimize  the 
estimated  error  (accuracy  of  the  model  on  unseen  cases).  Part  three  of  the  algorithm  applies  a  tree  pruning 
routine  which  eliminates  subtrees  of  a  node  if  the  estimation  error  is  higher  in  the  lower  branches  than  the 
estimation  error  when  using  the  node’s  internal  regression  model.  Finally,  a  smoothing  process,  with  the  goal 
of  improving  prediction  accuracy,  is  employed  to  ensure  that  the  linear  regression  models  of  adjacent  leaves  are 
continuous  and  smooth.  This  process  is  particularly  effective  when  some  of  the  linear  regression  models  are 
constructed  from  few  training  cases. 

3.1.1  Discussion 

Regression  trees  are  more  accurate  than  straightforward  linear  regression,  but  the  trees  are  often  cumbersome 
and  difficult  to  interpret  [14].  Model  trees  are  more  sophisticated  than  regression  trees  as  they  approximate 
continuous  functions  by  linear  “patches” — M5  model  trees  are  analogous  to  piecewise  linear  functions.  M5 
model  trees  have  an  advantage  over  regression  trees  with  respect  to  compactness  and  prediction  accuracy  due 
to  the  ability  of  model  trees  to  exploit  local  linearity  in  the  data.  Regression  trees  will  not  predict  values 
lying  outside  the  range  of  learned  training  cases,  while  M5  model  trees  can  extrapolate.  The  M5  model  tree 
algorithm  is  optimized  to  both  learn  known  cases  and  predict  unknown  cases.  The  trees  are  also  smaller,  easier- 
to-understand,  and  their  average  error  values  on  the  training  data  are  lower. 

M5  model  trees  tackle  the  difficulties  in  considering  a  multitude  of  cost  driving  factors;  they  determine  the 
right  set  of  independent  variables  (ship  attributes)  by  construction.  The  effect  of  the  M5  system  is  somewhat 
similar  to  the  use  of  indicator  variables  in  standard  linear  regression  analysis,  with  the  key  difference  being  that 
appropriate  indicators  are  identified  and  included  based  on  an  optimization  algorithm. 

M5  model  trees  have  been  shown  to  excel  even  when  limited  data  is  available  [15]  or  learn  efficiently  from 
large  data  sets.  They  can  handle  data  sets  which  include  systems  with  notable  differences,  missing  data,  and 
noise  (as  is  the  case  for  the  SAS-076  ship  data  set).  The  decision  tree  can  branch  on  any  variable  type:  nominal 
(e.g.,  military  vs.  non-military)  or  numeric  (e.g.,  tonnage  less  than  15000  or  greater  than  15000). 

The  use  of  M5  model  trees  for  numeric  prediction  has  increased  since  comprehensive  descriptions,  imple¬ 
mentations,  and  refinements  of  Quinlan’s  method  became  available  [15-19].  Recently  Chen  [20]  discussed  the 
benefits  of  the  system  for  estimating  the  cost  of  software  development. 

Neural  networks  are  commonly  used  for  predicting  numeric  quantities,  but  suffer  from  the  disadvantage 
of  producing  opaque  structures  that  mask  the  nature  of  the  solution.  Their  arbitrary  internal  representation 
means  that  there  can  be  variations  between  networks  trained  on  the  same  data.  In  comparison,  the  M5  system 
is  transparent  and  the  model  tree  construction  is  repeatable.  The  final  tree  and  linear  regression  models  are 
straightforward  and  clear. 

3.2  Application  to  SAS-076  Ship  Data  Set 

An  easy-to-use  implementation  of  Quinlan’s  M5  model  tree  is  available  as  part  of  the  WEKA  project  [21].  Wit¬ 
ten  and  Frank’s  textbook  [14]  provides  documentation  on  both  the  theory  and  implementation  (data  formatting, 
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execution,  etc.)  of  the  algorithm.  The  descriptive  attributes  (I.e),  (I.f),  (Il.h)  (as  per  Table  2),  the  complete  set 
of  technical  attributes,  and  a  normalized  cost  attribute  for  each  of  the  ships  in  the  SAS-076  data  set  was  used 
as  input  to  the  M5  model  tree  algorithm.  Using  WEKA10,  the  construction  of  the  M5  model  tree  for  the  ship 
database  of  57  ships  each  with  123  attributes  (701 1  elements)  took  less  than  a  second  of  computation  on  an 
Intel(R)  2.4GHz  computer  with  4GB  RAM. 

Figure  4  shows  the  resulting  M5  model  tree.  The  root  of  the  decision  tree  splits  the  ships  in  two  based 
on  the  number  of  air-cushioned  landing  craft  (LCAC)  that  the  ship  is  designed  to  carry  (it  should  be  noted 
that  this  split  also  groups  all  ships  void  of  a  well  deck).  Internal  nodes  of  the  tree  further  split  the  data  set  on 
attributes  such  as  the  number  (#)  of  torpedo  decoy  systems  on  board,  the  ship’s  rank  in  class,  the  maximum 
number  of  helicopters  supported,  the  ship’s  length,  and  the  ship’s  range  in  terms  of  total  distance  in  nautical 
miles.  The  tree  branches  out  to  nine  leaves  where  nine  corresponding  linear  regression  models  are  fitted.  The 
regression  models  are  presented  in  Table  4.  In  addition  to  the  ship  attributes  used  in  the  decision  tree  for 
branching,  the  linear  regression  models  use  the  ship’s  range  in  terms  of  total  sailing  time  in  hours.  The  linear 
regression  models  output  the  log-transformed  (decadic)  cost  of  a  ship.  The  individual  linear  regression  models 
are  mostly  intuitive:  the  cost  of  a  ship  increases  as  the  length  or  the  number  of  LCAC  supported  or  number  of 
torpedo  decoy  systems  increase(s).  The  regression  models  also  predict  a  shipbuilding  learning  curve  as  the  cost 
of  constructing  a  ship  decreases  as  a  function  of  the  ship’s  rank  in  class.  The  negative  coefficient  for  a  ship’s 
range  (sailing  time)  in  the  regression  models  is  counter-intuitive.  It  seems  unlikely  that  a  ship  will  cost  less  as 
its  range  increases.  The  SAS-076  ship  data  explains  this  anomaly:  the  median  ship  range  (sailing  time)  of  the 
ships  captured  in  the  SAS-076  ship  data  set  is  444  hours  and  the  mean  is  616  hours.  Only  6  of  the  57  ships  have 
a  range  greater  than  770  hours,  these  are  the  U.S.  Anchorage  class  LSDs  and  Sweden’s  Oden  icebreaker — their 
sailing  time  range  is  between  3-5  times  the  median.  The  Anchorage  class  LSD  costs  and  the  Oden  icebreaker 
cost  are  relatively  low  (in  comparison  to  the  other  SAS-076  ships).  The  combination  of  these  low  costing  ships 
and  outlying  sailing  time  ranges  provides  a  mathematical  explanation  for  the  negative  coefficient  of  the  sailing 
time  range  in  the  regression  models.  The  M5  model  can  be  potentially  adjusted  in  an  attempt  to  remove  such 
anomalies  by  disabling  the  particular  attribute  (sailing  time),  however  there  is  no  guarantee  that  the  regenerated 
model  will  not  substitute  this  attribute  with  another,  also  allocated  a  negative  coefficient.  Similarly,  removing 
the  instances  (e.g.,  Oden  and  Anchorage  class  LPDs)  from  the  data  set  provides  no  guarantees.  Rather  than 
subjectively  diminishing  the  data  set,  SAS-076  recommends  noting  the  anomalies  and  discussing  them  as  part 
of  the  results. 

Table  5  provides  the  minimum,  median,  mean,  and  maximum  values  found  in  the  SAS-076  ship  data  set  for 
the  attributes  used  by  the  M5  model.  Table  6  shows  the  classification  of  the  SAS-076  ships  by  the  M5  model 
tree. 

Figure  5  plots  the  actual  ship  costs  vs.  the  costs  predicted  by  the  M5  model  tree.  The  worth  of  a  regression- 
based  model  is  measured  by  the  coefficient  of  correlation,  the  quantity  that  gives  the  quality  of  a  least  squares 
fitting  to  the  original  data.  Let  x\  be  the  actual  cost  of  ship  i  and  y,  the  predicted  cost  (using  the  M5  model  tree), 
then  the  coefficient  of  correlation.  R,  of  the  M5  system  is  calculated  as 

nY.XiVi-Y.Xi'Lyi 

\J n'Lxf  -  (Lx;)2 yj, nl_y]  -  (L.V;)2 

where  n  =  57,  the  number  of  ships  in  the  data  set,  and  each  of  the  sums  are  over  these  59  ships.  By  design 
R  can  range  from  —1  to  +1  with  R  =  I  indicating  a  perfect  positive  linear  correlation.  A  correlation  greater 
than  0.8  is  generally  described  as  strong  [13].  The  value  R2,  known  as  the  coefficient  of  determination,  is  a 

1()The  program’s  default  parameters  for  the  M5  model  tree  algorithm  were  used. 
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#  of  LCAC  in  Well  Deck 


#  of  Torpedo  Decoys 


Rank  in  Class 

LM9  Maximum  #  of  Helicopters  Supported 


Figure  4:  M5  model  tree  applied  to  the  NATO  RTO  SAS  076  ship  data  set. 


measure  of  how  well  the  regression  line  represents  the  data — it  is  a  measure  determining  how  certain  one  can 
be  in  making  predictions  from  the  model.  R 2  is  also  the  ratio  of  the  explained  variation  (by  the  model)  to  the 
total  variation  of  the  data  set.  The  measures  of  worth  of  the  M5  model  tree  for  predicting  ship  costs  arc  a  strong 
R  =  0.96  and  R2  =  0.92.  The  coefficient  of  determination  indicates  that  92%  of  the  total  variation  in  the  ship 
costs  can  be  explained  by  the  linear  relationships  described  by  the  M5  model  tree  linear  regression  equations. 
The  remaining  8%  of  the  total  variation  remains  unexplained. 

Another  useful  indicator  is  the  mean  absolute  percent  error,  as  it  quantifies  the  the  amount  by  which  the 
estimated  cost  differs  from  the  actual  cost.  The  mean  absolute  percent  error  is  computed  as  as  follows: 


1  y  \xj-yj 
57  L  xt 


(2) 


The  mean  absolute  percent  error  for  the  M5  model  tree  applied  to  the  SAS-076  ship  data  set  is  12%.  The  stan¬ 
dard  deviation,  indicating  the  variation  that  M5  model  tree  predictions  have  from  the  actual  costs,  is  computed 
as 


N 


I  {xi-yi)2 

i=i _ 

57 


(3) 


for  a  value  of  46.4  million  NCC.  Mean  absolute  percent  errors  and  standard  deviations  specific  to  the  individual 
M5  model  tree  linear  regression  models  are  shown  in  Table  7.  For  the  puipose  of  determining  the  standard 
deviation  of  a  M5  model  tree  output  prediction,  the  standard  deviation  over  all  the  training  data  is  more  reflective 
of  the  M5  system  than  just  the  standard  deviation  of  the  training  cases  reaching  the  particular  leaf  node  used 
for  the  prediction.  By  M5  model  tree  construction,  each  of  the  training  cases  influence  the  structure  of  the  final 
model  tree. 
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Table  4:  M5  model  tree  linear  regression  models 


LM1 

LM2 

Log(Cost)  = 

7.4297 

Log(Cost)  = 

7.4208 

-  0.01 12  x  rank  in  class 

-  0.0112  x  rank  in  class 

+  0.0045  x  length  (m) 

+  0.0045  x  length  (m) 

-  0.0002  x  range  (sailing  time  in  hrs) 

-  0.0002  x  range  (sailing  time  in  hrs) 

+  0.0445  x  #  of  LCAC  in  well  deck 

+  0.0445  x  #  of  LCAC  in  well  deck 

+  0.1 104  x  #  of  torpedo  decoys 

+  0.1104  x  #  of  torpedo  decoys 

LM3 

LM4 

Log(Cost)  = 

7.6222 

Log(Cost)  = 

7.7567 

-  0.0167  x  rank  in  class 

-  0.0172  x  rank  in  class 

+  0.0041  x  length  (m) 

+  0.0032  x  length  (m) 

-  0.0002  x  range  (sailing  time  in  hrs) 

-  0.0002  x  range  (sailing  time  in  hrs) 

+  0.0445  x  #  of  LCAC  in  well  deck 

+  0.0445  x  #  of  LCAC  in  well  deck 

+  0.0659  x  #  of  torpedo  decoys 
+  0.00001  x  range  (distance  in  nm) 

+  0.0659  x  #  of  torpedo  decoys 

LM5 

LM6 

Log(Cost)  = 

7.7912 

Log(Cost)  = 

7.9846 

-  0.0170  x  rank  in  class 

-  0.0245  x  rank  in  class 

+  0.0030  x  length  (m) 

+  0.0038  x  length  (m) 

-  0.0002  x  range  (sailing  time  in  hrs) 

-  0.0003  x  range  (sailing  time  in  hrs) 

+  0.0445  x  #  of  LCAC  in  well  deck 

+  0.0300  x  #  of  LCAC  in  well  deck 

+  0.0659  x  #  of  torpedo  decoys 

+  0.0343  x  #  of  torpedo  decoys 

LM7 

LM8 

Log(Cost)  = 

8.1461 

Log(Cost)  = 

8.3575 

-  0.0361  x  rank  in  class 

-  0.0312  x  rank  in  class 

+  0.0035  x  length  (m) 

+  0.0024  x  length  (m) 

-  0.0003  x  range  (sailing  time  in  hrs) 

-  0.0003  x  range  (sailing  time  in  hrs) 

+  0.0300  x  #  of  LCAC  in  well  deck 

+  0.0300  x  #  of  LCAC  in  well  deck 

+  0.0343  x  #  of  torpedo  decoys 

+  0.0343  x  #  of  torpedo  decoys 

LM9 

Log(Cost)  = 

8.3001 

-  0.0221  x  rank  in  class 
+  0.0020  x  length  (m) 

-  0.0003  x  range  (sailing  time  in  hrs) 

+  0.0300  x  #  of  LCAC  in  well  deck 
+  0.0343  x  #  of  torpedo  decoys 
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Table  5:  Statistics  of  attributes  used  in  the  M5  model  tree  linear  regression  models. 


Attribute 

Minimum 

Median 

Mean 

Maximum 

Rank 

1 

3 

3.68 

12 

Length 

103.7 

173.8 

170.3 

203.4 

Range  (sailing  time) 

385 

444 

616 

2308 

#  LCAC 

0 

2 

2 

4 

#  Torpedo  decoys 

0 

0 

2 

8 

#  of  helicopters  supported 

0 

5 

6 

18 

Table  6:  M5  model  tree  classification  of  SAS-076  ships. 


LMl 

LM2 

LM3 

LM4 

LM5 

LM6 

LM7 

LM8 

LM9 

Svalbard 

Carlskrona 

Thomaston 

Plymouth  Rock 

Lyme  Bay 

Anchorage 

Whidbey  Island 

Raleigh 

Tortuga 

Protecteur 

Atle 

Largs  Bay 

Fort  Snelling 

Mounts  Bay 

Portland 

Germantown 

Vancouver 

Rushmore 

Preserver 

Oden 

Ocean 

Point  Defiance 

Cardigan  Bay 

Pensacola 

Fort  McHenry 

La  Salle 

Ashland 

Spiegel  Grove 

Mount  Vernon 

Gunston  Hall 

Mistral 

Denver 

Alamo 

Fort  Fisher 

Comstock 

Tonnerre 

Juneau 

Hermitage 

Harpers  Ferry 

Austin 

Dixmude  (BPC3) 

Coronado 

Monticello 

Carter  Hall 

Ogden 

Shreveport 

Siroco 

Oak  Hill 

Duluth 

Nashville 

Pearl  Harbour 

Cleveland 

Trenton 

Dubuque 

Albion 

Bulwark 

Ponce 

Table  7:  Mean  absolute  percent  errors  of  known  instances  and  standard  deviations  per  individual  M5  model  tree  linear  models. 


LMl 

LM2 

LM3 

LM4 

LM5 

LM6 

LM7 

LM8 

LM9 

Mean  %  error: 

22% 

27% 

17% 

3% 

33% 

12% 

14% 

8% 

6% 

Standard  deviation 

24.3M 

16.9M 

53.0M 

6.4M 

45.6M 

43.4M 

78.0M 

39.3M 

24.3M 

#  of  instances: 

3 

3 

3 

8 

3 

9 

12 

6 

10 
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Figure  5:  M5  model  tree:  correlation  plot  of  actual  vs.  predicted  ship  costs  (millions  NCC). 
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3.2.1  Comparison  to  Linear  Regression  Models 

The  best  CER  returned  by  applying  simple  linear  regression  on  the  SAS-076  data  set1 1  is 

Logio(cost )  =  6.95  +  0.01  x  length  of  ship  (in  meters),  (4) 

with  an  R  value  of  0.75  (R2  =  0.56).  Applying  multiple  linear  regression  with  a  greedy  attribute  selection 
method  (step  through  the  attributes  removing  the  one  with  the  smallest  standardized  coefficient  until  no  im¬ 
provement  is  observed  in  the  estimate  of  the  error  given  by  the  Akaike  information  criterion  [22]),  yields  the 
CER 


Logio(cost)  =  5.7368  —  0.0224  x 
+  0.0121  x 
+  0.0338  x 
+  0.1071  x 
-0.0001  x 
+  0.0012  x 

-  0.0876  x 

-  0.0239  x 


rank  in  class 

length  (in  meters) 

beam  (in  meters) 

draught  (in  meters) 

full  load  displacement  (in  tonnes) 

crew  size 

number  of  propeller  shafts 
number  of  guns  of  calibre  >75, 


(5) 


with  an  R  value  of  0.92  (R2  =  0.85).  While  the  CER  produced  by  applying  simple  linear  regression  is  straight¬ 
forward  to  understand,  the  negative  coefficient  signs  of  the  multiple  linear  regression  CER  makes  its  inter¬ 
pretation  non-trivial — detailed  analysis  of  the  input  data  is  required.  For  comparison  to  the  M5  model  tree, 
the  straightforward  linear  regression  estimates  for  HMS  Rotterdam  and  Johan  de  Witt  LPDs  are  presented  in 
Section  5.0. 


The  results  presented  in  Table  7  show  that  linear  regression  model  LM7  contributes  greatest  to  the  standard 
deviation.  Further  analysis  revealed  that  LM7  models  the  nine  highest  costing  ships,  all  over  400  million 
NCC.  In  eight  of  these  cases,  LM7  underestimates  the  actual  cost.  By  the  piece-wise  linear  M5  model  tree 
construction,  LM7  is  influenced  by  the  adjacent  linear  regression  models  LM6  and  LM8.  To  evaluate  the 
degree  of  this  influence,  a  separate  multiple  linear  regression  model  was  fitted  to  the  ships  reaching  the  LM7 
leaf  using  the  same  five  ship  attributes  as  LM7.  The  resulting  CER  is  as  follows: 

Logio(cost )  =  8.6376  —  0.0651  x  rank  in  class  (6) 

+  0.0637  x  number  of  LCAC. 

with  an  R  value  of  0.95  (R2  =  0.90),  mean  absolute  percent  error  of  7%  and  a  standard  deviation  of  34.8  million 
NCC.  This  result  indicates  that  it  is  indeed  possible  to  derive  better  models  for  subsets  of  the  data.  This  does  not 
depreciate  the  M5  model  tree  algorithm  whose  strength  is  its  optimization  in  predicting  unknown  cases  (rather 
than  simply  memorizing  the  known).  Using  the  linear  regression  model  of  Equation  6  in  place  of  LM7  would 
leave  adjacent  linear  models  LM6,  LM7,  and  LM8  sharply  discontinuous.  The  M5  system  applies  a  smoothing 
process  (see  Annex  ??)  to  construct  piece-wise  linear  regression  models12. 

1 1  Nominal  attributes  and  attributes  for  which  data  was  missing  was  omitted. 

12Experiments  by  Wang  and  Witten  [15]  show  that  this  smoothing  substantially  increases  the  accuracy  of  predictions  of  unseen  cases. 


B1  - 18 


RTO-TR-SAS-076 


ANNEX  B1  -  ESTIMATING  THE  ACQUISITION 
COST  OF  THE  ROYAL  NETHERLANDS  NAVY 
LANDING  PLATFORM  DOCK  SHIPS:  AN  EX  POST  ANALYSIS 


4.0  COST  ESTIMATION  BY  ANALOGY 

Cost  estimation  by  analogy  is  typically  accomplished  by  forecasting  the  cost  of  a  new  system  based  on  the 
historical  cost  of  similar  or  analogous  system  [23],  There  must  be  a  reasonable  correlation  between  the  new 
and  historical  system.  The  cost  of  the  historical  system  is  adjusted  by  undertaking  a  technical  evaluation  of  the 
differences  between  the  systems,  deducting  the  cost  of  components  that  are  not  comparable  to  the  new  design 
and  adding  estimated  costs  of  the  new  components.  Subject  matter  experts  are  required  to  make  a  subjective 
evaluation  of  the  differences  between  the  new  system  of  interest  and  the  historical  system.  Subjectively  chosen 
complexity  factors  are  often  used  to  adjust  the  analogous  system’s  cost  to  produce  an  estimate.  The  credibility 
of  the  estimate  for  the  new  system  may  be  undermined  if  the  adjustment  factors  are  not  substantiated — this  is  a 
key  disadvantage  of  the  traditional  analogy  method. 

In  the  next  section  hierarchical  cluster  analysis  is  used  for  a  novel  cost  estimation  by  analogy  approach  that 
is  void  of  the  subjectivity  inherent  (of  the  traditional  approach)  in  quantifying  the  cost  of  the  technical  and  other 
differences  between  the  historical  system  and  the  new  system.  The  approach  also  considers  multiple  analogous 
systems  rather  than  just  one. 

4.1  Hierarchical  Cluster  Analysis 

Hierarchical  cluster  analysis  is  a  data  mining  approach  that  facilitates  cost  estimation  by  analogy  by  identifying 
the  systems  that  are  the  “nearest  neighbours”  to  the  new  system.  Hierarchical  cluster  methods  produce  a  hier¬ 
archy  of  clusters  grouping  similar  items  together:  from  small  clusters  of  very  similar  items  to  large  clusters  that 
include  more  dissimilar  items.  In  particular,  agglomerative  hierarchical  methods  work  by  first  finding  the  clus¬ 
ters  of  the  most  similar  items  and  progressively  adding  less  similar  items  until  all  items  have  been  included  into 
a  single  large  cluster.  Hierarchical  agglomerative  cluster  analysis  begins  by  calculating  a  matrix  of  distances 
among  systems  expressing  all  possible  pairwise  distances  among  them.  Initially  each  system  is  considered  a 
group,  albeit  of  a  single  item.  Clustering  begins  by  finding  the  two  systems  that  are  most  similar,  based  on 
the  distance  matrix,  and  merging  them  into  a  single  group.  The  characteristics  of  this  new  group  are  based 
on  a  combination  of  the  systems  in  that  group.  This  procedure  of  combining  two  groups  and  merging  their 
characteristics  is  repeated  until  all  the  systems  have  been  joined  into  a  single  large  cluster. 

Hierarchical  cluster  analysis  is  a  useful  means  of  observing  the  structure  of  the  data  set.  The  results  of  the 
cluster  analysis  are  shown  by  a  dendrogram  (tree),  which  lists  all  of  the  samples  and  indicates  at  what  level  of 
similarity  any  two  clusters  were  joined.  The  x-axis  is  a  measure  of  the  similarity  or  distance  at  which  clusters 
join.  The  resulting  clustering  can  be  used  to  estimate  the  cost  of  a  new  system  by  taking  a  weighted  average  of 
the  cost  of  historical  systems  based  on  the  relative  distances  between  the  new  system  and  the  historical  systems. 

4.2  Application  to  SAS-076  Ship  Data  Set 

Using  the  SAS-076  ship  data  set,  hierarchical  clustering  is  used  to  define  a  ship  distance  function  which  takes 
as  input  ship  attributes  for  a  pair  ships  and  outputs  a  single  value  indicating  the  distance,  or  similarity,  between 
the  two  ships.  Formally,  define 

dijk  =  distance  between  ship  i  and  j  with  respect  to  attribute  k.  (7) 

For  numeric  attributes,  cljjit  is  normalized  to  lie  in  the  [0, 1]  range  with  d;jk  =  1  indicating  that  ships  i  and  j  lie 
at  opposite  ends  of  the  observed  spectrum  for  attribute  k  (e.g.,  shortest  and  longest  length  ships),  and  dijk  =  0 
indicating  that  the  ships  are  the  same  with  respect  to  attribute  k.  For  nominal  attributes,  dijk  is  binary — set  to  0 
if  ship  i  and  j  are  the  same  with  respect  to  attribute  k,  and  1  if  they  are  not. 
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A  variety  of  distance  metrics  can  be  used  to  calculate  similarity  of  two  ships  based  on  the  attribute  distances 
dijk-  Using  a  simple  Euclidean  distance  metric,  the  aggregate  distance  between  two  ships  i  and  j,  is  expressed 
as 


where  A  is  the  subset  of  attributes  considered.  Figure  7  illustrates  the  dendrogram  resulting  from  the  hierarchi¬ 
cal  clustering  of  ships  using  a  the  simple  distance  function  (equation  (8))  and  a  set  of  123  attributes  including 
descriptive  attributes  (I.e),  (I.f),  (Il.h)  and  the  complete  set  of  technical  attributes  as  per  Table  2.  The  dendro¬ 
gram  indicates  that  HMS  Rotterdam  is  best  grouped  with  HMS  Johan  de  Witt.  Both  Netherlands  LPDs  are  then 
clustered  with  the  French  Mistral  class  AASs,  namely  the  Mistral,  Tonnerre,  and  Dixmude.  The  next  closest 
ship  to  this  agglomerative  cluster  is  the  United  Kingdom’s  Ocean  FPH,  and  so  on. 

Computing  the  distances  between  all  pairs  of  ships  using  equation  (8),  the  cost  of  a  ship  i  can  be  estimated 
by  computing  the  weighted-average  cost  of  the  other  ships.  Fet  Cj  be  the  known  cost  of  ship  j,  then 


C-Y0! 

_  L  d2 

m  % 


(9) 


is  the  estimated  cost  of  ship  i.  Figure  6  plots  the  actual  ship  costs,  Q,  vs.  the  costs  predicted,  C,-,  by  the  analogy 
method  using  hierarchical  clustering  based  on  a  simple  distance  metric.  The  measures  of  worth  of  the  analogy 
method  via  hierarchical  clustering  analysis  (simple  distance  metric)  for  predicting  ship  costs  are  R  =  0.48 
(coefficient  of  correlation  as  calculated  by  Equation  1)  and  R 2  =  0.23.  The  mean  absolute  percent  error  is  49% 
and  the  standard  deviation  of  1 12  million  NCC.  The  approach  does  poorly  in  learning  the  known  ship  costs. 


Figure  6:  Hierarchical  clustering  with  simple  distance  function:  correlation  plot  of  actual  vs.  predicted  ship  costs  (millions  NCC). 
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Figure  7:  Dendrogram  illustrating  the  arrangement  of  the  clusters  produced  by  the  hierarchical  clustering  of  ships  (simple  distance  function). 
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ORGANIZATION 


An  assumption  in  the  above  approach  is  that  all  attributes  are  of  equal  importance;  the  (normalized)  dif¬ 
ferences  or  similarities  for  each  attribute  contribute  equally  to  the  measure  of  similarity  between  ships.  To 
potentially  improve  the  predictive  capability  of  the  method,  attribute  weights  are  defined.  Let 

Wk=  weight  of  attribute  k.  (10) 


Using  a  weighted  Euclidean  distance  metric,  the  aggregate  distance  between  two  ships  i  and  j,  is  expressed  as 


where  £  ny  =  1  and  wy  >  0  for  all  k.  As  before,  let  Cj  be  the  known  cost  of  ship  j,  then 

keA 


mdi] 


1 


£  J2 

m  ij 


(12) 


is  the  estimated  cost  of  ship  i  using  weighted  attributes.  The  optimal  allocation  of  weights  is  determined  by 
minimizing  the  prediction  error  for  the  known  ships, 


minimize 


E(c,-c,)2. 


(13) 


The  resulting  mathematical  optimization  is  a  non-linear  convex  program.  With  the  full  set  of  attributes  used 
previously,  |A|  =  123,  the  mathematical  program  was  too  computationally  intensive  to  solve  in  reasonable  time 
using  Wolfram  Mathematica©  on  a  Intel(R)  2.4GHz  computer  with  4GB  RAM.  To  reduce  the  dimensional¬ 
ity  of  the  problem,  a  smaller  subset  of  attributes  was  selected.  While  there  are  numerous  attribute  selection 
algorithms  (see  [14]),  principal  component  analysis  (PCA),  a  tool  in  exploratory  data  analysis  and  for  mak¬ 
ing  predictive  models,  was  used.  PCA  involves  a  mathematical  procedure  that  transforms  a  number  of  possibly 
correlated  variables  into  a  smaller  number  of  uncorrelated  variables  called  principal  components.  The  first  prin¬ 
cipal  component  accounts  for  as  much  of  the  variability  in  the  data  as  possible,  and  each  succeeding  component 
accounts  for  as  much  of  the  remaining  variability  as  possible13. 

The  WEKA  software  tool  [21]  was  used  to  perform  PCA.  The  ship  data  set  consisting  of  123  attributes 
(as  used  with  the  simple  distance  function)  was  reduced  to  a  set  of  16  macro-attributes  accounting  for  95%  of 
the  original  set’s  variability.  Table  8  lists  the  macro-attributes  and  the  respective  percentage  of  data  variability 
(cumulative)  each  accounts  for. 

Each  macro-attribute  is  a  linear  combination  of  the  original  attributes.  For  example,  macro-attribute  A2  is 


13PCA  involves  the  calculation  of  the  eigenvalue  decomposition  of  a  data  covariance  matrix.  Dimensionality  reduction  is  accom¬ 
plished  by  choosing  enough  eigenvectors  to  account  for  some  percentage  of  the  variance  in  the  original  data.  PCA  can  be  thought  of  as 
revealing  the  internal  structure  of  the  data  in  a  way  which  best  explains  the  variance  in  the  data.  If  a  multivariate  dataset  is  visualized 
as  a  set  of  coordinates  in  a  high-dimensional  data  space  (1  axis  per  variable),  PCA  supplies  the  user  with  a  lower-dimensional  picture, 
a  “shadow”  of  this  object  when  viewed  from  its  (in  some  sense)  most  informative  viewpoint. 
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Table  8:  Principal  component  analysis  results 


Macro-Attribute 

%  of  Data  Variability  Accounted  for 
Proportion  Cumulative 

A1 

17% 

17% 

A2 

12% 

29% 

A3 

11% 

41% 

A4 

9% 

49% 

A5 

8% 

57% 

A6 

7% 

64% 

A7 

6% 

69% 

A8 

5% 

74% 

A9 

4% 

78% 

A10 

3% 

81% 

All 

3% 

85% 

A12 

3% 

88% 

A13 

3% 

90% 

A14 

2% 

92% 

A15 

2% 

94% 

A16 

1% 

95% 

A2  =  0.204  x  length  (14) 

+  0. 196  X  beam  width 
+  0.183  x  vehicle  space 
+  0.18  x  #  of  expeditionary  fighting  vehicles 
+  0. 165  x  1  if  has  a  well  deck,  otherwise  0 
+  0.165  x  width  of  the  well  deck 
+  0. 164  x  length  of  the  well  deck 
+  0. 159  x  #  of  large  personnel  landing  craft 
+  0.156  X  #  of  Chinook  helicopters  supported 
+  0. 155  x  full  load  displacement 
+  0.153  X  #  of  combat  data  systems 
+  0.150  x  light  load  displacement 
+  0.144  x  well  deck  capacity 
+  0. 143  x  #  of  elevators 
+  0.142  X  vehicle  fuel  capacity 
etc. 
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ORGANIZATION 


(Only  the  top  15  attributes — in  terms  of  PCA  coefficient  size — are  enumerated  in  Equation  14.)  Attempting  to 
find  solutions  to  the  mathematical  program  (13)  with  |A|  =  16  macro-attributes  resulted  in  memory  overflow. 
Using  the  top  ten  macro-attributes,  accounting  for  80%  of  the  original  data  set’s  variance,  an  optimal  solution 
for  the  macro-attribute  weights  was  determined.  The  weights  are  listed  in  Table  9.  Only  macro-attributes 
Al,  A2,  A4,  and  A8  have  non-zero  weights.  This  is  a  typical  extreme  output  of  mathematical  optimization 
software.  There  may  exist  other  optimal  solutions  with  other  non-zero  macro-attributes  weights.  Annex  ?? 
shows  the  resulting  top  ten  macro-attributes  for  the  57  SAS-076  ships  as  well  for  HMS  Rotterdam  and  Johan 
de  Witt  LPDs. 


Table  9:  Optimal  macro-attribute  weights  for  cost  estimation  by  hierarchical  clustering 


Attribute 

Weight 

Al 

0 

A2 

0.452 

A3 

0 

A4 

0 

A5 

0.334 

A6 

0 

A7 

0 

A8 

0 

A9 

0 

A10 

0.214 

The  hierarchical  cluster  analysis  using  the  weighted  distance  function  on  the  top  ten  macro- attributes  deter¬ 
mined  by  PCA  is  visualized  in  Figure  8.  The  figure  illustrates  the  resulting  dendrogram  indicating  that  HMS 
Rotterdam  LPD  is  grouped  with  the  United  Kingdom’s  Albion  class  LPDs  and  Largs  Bay  class  LSDs,  followed 
by  France’s  Siroco  LSD,  etc.  HMS  Johan  de  Witt  LPD  is  clustered  with  Sweden’s  Svalbard  icebreaker,  France’s 
Mistral  class  AAS,  United  Kingdom’s  Ocean  LPH,  and  so  on14. 


14As  expected,  ships  within  the  same  class  (but  different  rank)  are  closely  grouped  together. 
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Figure  8:  Dendrogram  illustrating  the  arrangement  of  the  clusters  produced  by  the  hierarchical  clustering  of  ships  (weighted  distance  function). 
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ORGANIZATION 


Figure  9  plots  the  actual  ship  costs,  Q,  vs.  the  costs  predicted,  Q,  by  the  analogy  method  using  hierarchical 
clustering  based  on  a  weighted  distance  metric.  The  measures  of  worth  of  the  analogy  method  via  hierarchical 
clustering  analysis  (weighted  distance  matrix)  for  predicting  ship  costs  are  R  =  0.93  (coefficient  of  correlation 
as  calculated  by  Equation  1)  and  R 2  =  0.86.  The  latter  coefficient  of  determination  indicates  that  86%  of 
the  total  variation  in  the  ship  costs  can  be  explained  by  an  average  cost  of  the  known  ships  weighted  by  an 
optimized  distance  metric.  The  mean  absolute  percent  error  is  16%  and  the  standard  deviation  is  55.9  million 
NCC — improvements  over  the  hierarchical  clustering  based  on  the  simple  distance  metric. 


Actual  Cost  (millions  NCC) 

Figure  9:  Weighted  hierarchical  clustering:  correlation  plot  of  actual  vs.  predicted  ship  costs  (millions  of  NCC). 


4.3  Discussion 

Figure  9  shows  that  the  analogy  method  using  hierarchical  clustering  based  on  a  weighted  distance  metric 
underestimates  the  cost  of  seven  of  the  eight  most  expensive  ships  (all  over  406  million  NCC).  This  was  also 
a  characteristic  of  the  parametric  estimation  presented  in  Section  3.2.  In  the  latter  it  was  conjectured  that  the 
underestimation  was  likely  a  result  of  the  smoothing  and  pruning  functions  of  the  M5  model  tree  algorithm, 
designed  to  optimize  the  predictive  capability  of  the  method.  However,  the  underestimation  of  expensive  ships 
in  the  second,  independent  method,  are  potentially  an  indication  that  the  attributes  (and  their  values)  of  the 
SAS-076  ship  data  set  do  not  provide  enough  information  to  help  distinguish  the  highest  costing  ships  from 
their  peers. 
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5.0  RESULTS 

The  results  of  the  M5  model  tree  system  are  described  in  Subsection  5.1.  The  point  estimates  obtained  using  the 
simple  linear  regression  (Equation  4)  and  multiple  linear  regression  (Equation  5)  CER  models  are  revealed  in 
Subsection  5.2.  Subsection  5.3  details  the  results  of  the  hierarchical  clustering  method.  Subsection  5.4  discusses 
which  result  should  be  considered  the  primary  estimate,  and  in  Subsection  5.5  the  results  of  both  models  are 
are  compared  with  the  actual  production  and  development  costs  (revealed  post  estimation). 

5.1  M5  Model  Tree  Results 

The  technical  specifications  of  HMS  Rotterdam  LPD  (Table  3)  were  used  to  trace  down  the  M5  model  tree 
depicted  in  Figure  4.  In  particular,  the  input  data  indicates  that  HMS  Rotterdam  LPD  does  not  have  the  capacity 
to  carry  LCAC  in  its  well  deck,  carries  one  torpedo  decoy  system,  and  is  ranked  first  in  class.  This  results  in 
linear  regression  model  LM3  as  per  Table  4.  Similarly,  the  attributes  of  HMS  Johan  de  Witt  LPD  are  used 
to  follow  the  same  path  in  the  M5  model  tree  to  linear  regression  model  LM3.  The  linear  regression  model 
LM3  has  the  ship’s  rank  in  class,  length,  range  in  terms  of  sailing  time  in  hours  and  distance  (nm),  number  of 
LCAC,  and  number  of  torpedo  decoy  systems  as  independent  variables.  Using  the  LM3  model,  the  predicted 
development  and  production  cost  of  HMS  Rotterdam  LPD  is  197.7  million  NCC.  The  predicted  cost  of  HMS 
Johan  de  Witt  LPD  is  212.3  million  NCC15. 

Figures  10  and  1 1  illustrate  the  log-normal  probability  density  and  cumulative  distribution  functions  for  the 
M5  model  tree  estimates  for  the  cost  of  HMS  Rotterdam  LPD  and  HMS  Johan  de  Witt  LPD. 


Figure  10:  Probability  density  function  (a)  and  cumulative  distribution  function  (b)  of  the  M5  model  tree  estimate  of  HMS  Rotter¬ 
dam  LPD  cost. 


Table  10  lists  the  predicted  costs  for  incremental  percentiles  of  the  fitted  log-normal  probability  density 
functions.  The  predicted  costs  coincide  to  the  50th  percentile  of  the  respective  log-normal  probability  distrib¬ 
ution  functions.  This  effect  is  explained  by  Goldberger  [24] :  when  a  power-function  form  is  used  for  a  CER, 
attention  shifts  from  the  mean  to  the  median  as  a  measure  of  central  tendency;  the  CER  yields  an  estimate  of  the 

l5HMS  Rotterdam  LPD  and  HMS  Johan  de  Witt  LPD  differ  in  range  (distance  and  sailing  time  in  hours)  and  length — two  of  the 
independent  variables  in  LM3. 
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Figure  1 1 :  Probability  density  function  (a)  and  cumulative  distribution  function  (b)  of  the  M5  model  tree  estimate  of  HMS  Johan 
de  Witt  LPD  cost. 


median  value  of  Y  rather  than  the  mean.  The  mean  of  the  presented  log-normal  distributions  are  200.2  million 
NCC  and  215.0  million  NCC  for  HMS  Rotterdam  LPD  and  HMS  Johan  de  Witt  LPD  respectively. 
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Table  10:  Percentiles  of  the  fitted  log-normal  density  functions  for  the  M5  model  tree  estimated  HMS  Rotterdam  LPD  and  HMS 
Johan  de  Witt  LPD  costs  (millions  NCC). 


Percentile 

HMS  Rotterdam 

HMS  Johan  de  Witt 

0.05 

152.2 

163.5 

0.10 

161.3 

173.2 

0.15 

167.7 

180.1 

0.20 

173.0 

185.7 

0.25 

177.6 

190.7 

0.30 

181.9 

195.3 

0.35 

186.0 

199.7 

0.40 

189.9 

203.9 

0.45 

193.8 

208.1 

0.50 

197.7 

212.3 

0.55 

201.7 

216.6 

0.60 

205.8 

221.0 

0.65 

210.2 

225.7 

0.70 

214.9 

230.7 

0.75 

220.1 

236.3 

0.80 

226.0 

242.7 

0.85 

233.1 

250.3 

0.90 

242.3 

260.2 

0.95 

256.7 

275.7 

5.2  Linear  Regression  Results 

Using  simple  linear  regression  CER,  Equation  4,  the  estimate  for  HMS  Rotterdam  LPD  is  219.2  million  NCC. 
The  estimate  for  HMS  Johan  de  Witt  LPD  is  289.7  million  NCC.  Using  the  multiple  linear  regression  CER, 
Equation  5,  the  estimate  for  HMS  Rotterdam  LPD  is  158.9  million  NCC.  The  estimate  for  HMS  Johan  de  Witt 
LPD  is  201.4  million  NCC. 

5.3  Hierarchical  Cluster  Analysis  Results 

The  technical  specifications  of  HMS  Rotterdam  LPD  (Table  3)  were  mapped  to  the  ten  attributes  selected  by 
the  principal  component  analysis  (see  Annex  ??).  Using  the  optimized  macro-attribute  weights  determined  in 
Section  4.2,  the  normalized  relative  distances  of  HMS  Rotterdam  LPD  to  the  other  ships  are  listed  in  Table  1 1 
(the  distances  have  been  normalized  so  that  the  furthest  ship  has  a  distance  of  1).  The  resulting  hierarchical 
clustering  cost  estimate  for  HMS  Rotterdam  LPD  is  214.6  million  NCC,  and  243.9  million  NCC  for  HMS 
Johan  de  Witt  LPD. 

Figures  12  and  13  illustrate  the  log-normal  probability  density  and  cumulative  distribution  functions  for  the 
hierarchical  cluster  estimates  for  the  cost  of  HMS  Rotterdam  LPD  and  HMS  Johan  de  Witt  LPD.  Table  12  lists 
the  predicted  costs  for  incremental  percentiles  of  the  fitted  log-normal  probability  density  functions.  The  mean 
of  the  presented  log-normal  distributions  are  219.8  million  NCC  and  249.8  million  NCC  for  HMS  Rotterdam 
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Table  11 :  Weighted  distance  of  the  Rotterdam  LPD  to  ships  in  the  Rotterdam  data  set. 


Name 

Distance 

Name 

Distance 

Name 

Distance 

Rotterdam 

0.000 

Vancouver 

0.312 

Nashville 

0.639 

Largs  Bay 

0.034 

La  Salle 

0.316 

Trenton 

0.646 

Lyme  Bay 

0.034 

Harpers  Ferry 

0.405 

Ponce 

0.650 

Mounts  Bay 

0.035 

Carter  Hall 

0.431 

Whidbey  Island 

0.655 

Cardigan  Bay 

0.035 

Oak  Hill 

0.435 

Germantown 

0.659 

Oden 

0.039 

Pearl  Harbour 

0.439 

Fort  McHenry 

0.664 

Carlskrona 

0.044 

Anchorage 

0.546 

Gunston  Hall 

0.668 

Johan  de  Witt 

0.046 

Portland 

0.550 

Comstock 

0.673 

Atle 

0.052 

Pensacola 

0.553 

Tortuga 

0.678 

Albion 

0.057 

Mount  Vernon 

0.557 

Rushmore 

0.682 

Bulwark 

0.058 

Fort  Fisher 

0.561 

Ashland 

0.687 

Siroco 

0.067 

Austin 

0.601 

Thomaston 

0.971 

Svalbard 

0.068 

Ogden 

0.606 

Plymouth  Rock 

0.975 
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Figure  12:  Probability  density  function  (a)  and  cumulative  distribution  function  (b)  of  the  hierarchical  clustering  estimate  of  HMS 
Rotterdam  LPD  cost. 
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Figure  13:  Probability  density  function  (a)  and  cumulative  distribution  function  (b)  of  the  hierarchical  clustering  estimate  of  HMS 
Johan  de  Witt  LPD  cost. 


Table  12:  Percentiles  of  the  fitted  log-normal  density  functions  for  the  hierarchical  clustering  estimated  HMS  Rotterdam  LPD  and 
HMS  Johan  de  Witt  LPD  costs  (millions  NCC). 


Percentile 

HMS  Rotterdam 

HMS  Johan  de  Witt 

0.05 

143.1 

170.4 

0.10 

153.7 

184.5 

0.15 

161.3 

194.6 

0.20 

167.6 

203.0 

0.25 

173.2 

210.6 

0.30 

178.4 

217.6 

0.35 

183.3 

224.3 

0.40 

188.1 

230.8 

0.45 

192.9 

237.3 

0.50 

197.7 

243.9 

0.55 

202.6 

250.7 

0.60 

207.8 

257.7 

0.65 

213.2 

265.3 

0.70 

219.1 

273.4 

0.75 

225.7 

282.5 

0.80 

233.2 

293.0 

0.85 

242.3 

305.7 

0.90 

254.3 

322.5 

0.95 

273.1 

349.1 
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5.4  Discussion 

The  estimates  generated  by  the  M5  model  tree  are  considered  to  be  the  primary  estimates  for  HMS  Rotterdam 
and  Johan  de  Witt,  the  hierarchical  clustering  estimates  are  considered  as  secondary  estimates.  This  decision 
was  driven  by  the  following: 

•  The  M5  model  tree  algorithms  are  optimized  to  both  learn  known  cases  and  predict  unknown  cases.  The 
attribute  weights  used  in  the  hierarchical  clustering  method  are  optimized  to  learn  the  known  cases. 

•  The  hierarchical  clustering  approach  uses  principal  component  analysis  to  reduce  the  dimensionality  of 
the  attribute  space.  Due  to  computational  limitations,  the  weight  optimization  method  could  only  be 
applied  on  the  top  ten  macro-attributes,  accounting  for  80%  of  the  original  data  set’s  variability.  In 
comparison,  the  M5  model  tree  algorithm  is  computationally  superior  as  it  efficiently  learns  from  large 
data  sets. 

•  The  M5  model  tree  results  in  a  better  correlation  measure,  lower  mean  absolute  percent  error,  and  smaller 
standard  deviation  in  estimating  the  known  cases. 

Table  13  synthesizes  the  predictions  and  compares  properties  of  the  M5  model  tree  and  hierarchical  clus¬ 
tering  methods. 
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Table  13:  Comparison  of  the  M5  model  tree  and  hierarchical  clustering  methods  and  their  estimates. 


M5  model  tree  Hierarchical  clustering 

HMS  Rotterdam  estimate 

197.7M  NCC 

214.6M  NCC 

HMS  Johan  de  Witt  estimate 

212.3M  NCC 

243. 9M  NCC 

Coefficient  of  correlation 

0.96 

0.93 

Coefficient  of  determination 

0.92 

0.86 

Standard  deviation 

46.4M  NCC 

55.9M  NCC 

Mean  absolute  %  error 

11% 

16% 

Ability  to  learn  known  cases 

/ 

/ 

Optimized  to  predict  unknown  cases 

/ 

X 

Uses  entire  data  set 

/ 

X 

In  Subsections  5.1  and  5.3,  the  cost  estimates  outputted  by  the  M5  system  and  the  hierarchical  clustering 
method  were  fitted  to  log-normal  distributions.  The  probability  distributions  of  the  cost  and  the  associated 
percentile  breakdowns  provide  decision  makers  the  ability  to  choose  a  budget  based  on  a  risk  tolerance.  The 
risk-averse  decision  maker  will  choose  a  higher  budget  amount  to  minimize  the  budget  risk — the  probability 
that  the  actual  cost  of  a  weapon  system  acquisition  program  will  end  up  exceeding  a  given  budget. 

5.5  Ex  Post  Revelation 

The  Royal  Netherlands  Navy  revealed  the  actual  development  and  production  costs  of  HMS  Rotterdam  and 
Johan  de  Witt  LPDs  to  the  NATO  RTO  SAS  076  Task  Group  once  the  cost  estimates  were  established.  The 
costs  were  then  normalized  to  the  fictitious  notional  common  currency.  Figures  14  and  15  illustrate  where 
the  actual  costs  (thick  vertical  lines)  fall  with  respect  to  the  log-normal  probability  density  and  cumulative 
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distribution  functions  for  the  M5  model  tree  and  hierarchical  cluster  cost  estimates  for  HMS  Rotterdam  LPD 
and  Johan  de  Witt  LPD.  Respecting  the  wishes  of  the  Netherlands,  even  the  exact  fictitious  costs  are  not  made 
more  explicit. 


Figure  14:  Probability  density  function  (a)  and  cumulative  distribution  function  (b)  of  the  M5  model  tree  (blue)  and  hierarchical 
clustering  (red)  estimates  of  HMS  Rotterdam  LPD  cost. 


Figure  15:  Probability  density  function  (a)  and  cumulative  distribution  function  (b)  of  the  M5  model  tree  (blue)  and  hierarchical 
clustering  (red)  estimates  of  HMS  Johan  de  Witt  LPD  cost. 


Table  14  compares  the  actual  development  and  production  costs  of  HMS  Rotterdam  and  HMS  Johan  de 
Witt  to  the  estimates  generated  by  the  M5  model  tree  and  hierarchical  clustering  methods.  The  percent  error  of 
the  M5  model  tree  estimates  (50th  percentile)  relative  to  the  actual  costs  are  -2%  (under  estimated)  for  HMS 
Rotterdam  LPD  and  -16%  (under  estimated)  for  HMS  Johan  de  Witt  LPD.  The  percent  error  of  the  hierarchical 
clustering  estimates  (50th  percentile)  relative  to  the  actual  costs  are  6%  (over  estimated)  for  HMS  Rotterdam 
LPD  and  -4%  (under  estimated)  for  HMS  Johan  de  Witt  LPD.  With  respect  to  the  fitted  log-normal  distributions, 
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Table  14:  Comparison  of  actual  to  estimated  costs  (millions  NCC). 


M5  model  tree  estimate 

%  error 

Hierarchical  clustering  estimate 

%  error 

HMS  Rotterdam 

197.7 

-2% 

214.6 

6% 

HMS  Johan  de  Witt 

212.3 

-16% 

243.9 

-4% 

the  actual  costs  lie  at  the  55th  (HMS  Rotterdam  LPD)  and  80th  (HMS  Johan  de  Witt  LPD)  percentiles  for  the 
M5  model  tree  results,  and  at  the  39th  (HMS  Rotterdam  LPD)  and  57th  (HMS  Johan  de  Witt  LPD)  percentiles 
for  the  hierarchical  clustering  results. 

In  retrospect,  it  is  interesting  to  recall  the  discussion  of  Section  3.2  on  the  negative  coefficient  of  a  ship's 
sailing  range  in  the  M5  model  tree’s  linear  regression  models.  It  was  noted  that  the  majority  of  the  ships  had 
a  range  under  770  hours,  the  impact  on  the  estimated  cost  of  these  ships  would  be  minimal  while  lowering 
estimates  of  ships  with  outlying  sailing  ranges  (over  1200  hours).  HMS  Johan  de  Witt  LPD’s  sailing  range 
is  listed  as  833  hours.  Substituting  the  median  sailing  range  (444  hours)  of  the  SAS-076  data  set,  effectively 
neutralizing  the  attribute,  would  result  in  a  revised  estimate  of  253.9  million  NCC,  which  is  within  1%  of  the 
actual  cost.  The  M5  model  tree  estimate  of  HMS  Rotterdam  LPD  less  sensitive  to  this  factor  as  its  sailing 
range  is  500  hours,  already  quite  close  to  the  median  of  444  hours,  neutralizing  the  attribute  by  substituting  the 
median  sailing  range  results  in  a  revised  estimate  of  202.8  million. 

6.0  LESSONS  IDENTIFIED 

This  section  intends  to  capture  the  feedback  and  lessons  identified  on  estimating  acquisitions  costs  of  the 
Netherlands'  Landing  Platform  Dock  Ships  Rotterdam  and  Johan  de  Witt.  The  section  is  divided  in  six  parts: 
general  management  of  the  estimating  process;  aims  and  cost  boundary:  data  and  assumptions;  methods,  models 
and  tools;  risk  and  uncertainty;  and  analysis  and  presentation  of  results. 

6.1  General  Management  of  the  Estimating  Process 

Initial  Plan 

SAS-076  started  with  a  plan  describing  the  activities  and  captured  this  in  a  Project  Initiation  Document  (PID). 
Originally  it  was  described  as  follows: 

For  the  acquisition  stage  a  top-down  approach  will  be  adopted,  using  the  information  in  the 
DADD  and  applying  the  parametric  method.  As  a  second  estimating  method  it  is  possible  to  apply 
the  analogy  method  using  public  available  data  (Internet,  NAO  ( UK)  report,  etc). 

Initially  the  cost-estimating  approach  consisted  of  two  iterations.  The  first  iteration  was  to  estimate  high- 
level  cost  elements  of  the  cost  breakdown  structure.  This  would  result  in  many  assumptions;  these  assumptions 
would  need  to  be  checked  by  the  Royal  Netherlands  (RNL)  Navy.  It  would  have  been  used  to  identify  cost 
drivers.  The  second  iteration  was  to  go  into  more  details  for  the  costs  drivers,  if  additional  data  was  available. 
The  main  milestones  for  the  first  iteration  were: 

•  Milestone  1:  Agreement  on  cost  elements  to  be  calculated  -  16/01/2009. 
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•  Milestone  2:  Release  of  the  data  and  assumptions  list  -  26/06/2009. 

•  Milestone  3:  Release  of  first  report  -  22/01/2010. 

The  main  Milestones  for  the  second  iteration  were: 

•  Milestone  4:  Agreement  on  new  cost  elements  to  be  calculated  -  22/01/2010. 

•  Milestone  5:  Release  of  the  updated  data  and  assumptions  list  -  09/07/2010. 

•  Milestone  6:  Release  of  second  iteration  report  -  01/10/2010. 


Actual  Implementation  of  the  Initial  Plan 

The  following  steps  were  actually  followed  after  having  established  the  PID: 

•  Agreement  on  the  Cost  Breakdown  Structure  (CBS):  follow  ANEP  41  (SWBS)  and  the  CBS  from  SAS- 
054  report; 

•  Developed  a  Data  and  Assumptions  Document  Definition  (DADD)  based  on  the  data  found  in  public 
sources,  participants’  inputs  and  the  RNLN; 

•  Investigated  software  tools  to  be  used;  and, 

•  Decision  on  methods  to  be  used:  analogy  and  parametric. 

Canadian  participants  proposed  to  use  a  data  mining  parametric  method  based  on  model  regression  trees 
and  a  new  analogy  method  based  on  clustering  analysis.  The  proposal  stemmed  from  SAS-054  Section  4.4 
recommendation  that  “The  best  cost  estimating  method  is  one  that  makes  the  best  use  of  the  data  available.  It 
is  therefore  recommended  to  employ  a  method  that  will  provide  as  much  detail  as  the  availability  of  the  input 
data  will  allow.  Therefore,  the  availability  of  data  is  a  major  factor  in  the  estimator’s  choice  of  estimating 
method.”  The  team  decided  to  follow  the  Canadian  proposal  and  to  deviate  from  the  original  plan.  By  adopting 
this  approach  the  second  iteration  was  considered  not  applicable  any  more.  In  addition,  the  SAS-076  started 
to  write  the  report  when  the  first  decisions  were  made  and  when  the  cost  estimation  models  were  developed. 
During  the  estimating  process  SAS-076  decided  not  to  further  develop  the  DADD  as  a  separate  document.  Data 
used  and  the  assumptions  made  are  included  in  the  final  report. 

Lessons  Identified 

The  group  did  not  work  continuously  on  this  project,  causing  additional  challenges  in  the  estimates  and  in 
the  management  of  some  important  issues.  The  SAS-076  group  considered  the  estimating  guidelines  and  the 
tasks  described  in  SAS-054  of  the  management  process  to  be  useful.  The  group  tried  to  follow  the  SAS  054 
guidelines  the  work  as  much  as  possible,  however  the  group  was  restricted  by  the  limited  available  resources 
and  competence  available  in  the  group  or  available  within  the  group  members  professional  network.  This 
affected,  in  a  certain  extent,  the  work  process.  The  conclusion  is  that  following  the  SAS  054  guidelines  could 
be  followed,  however  it  takes  some  time  and  resources.  Therefore  the  resources  and  the  budget  required  to  do 
an  estimate  according  to  the  SAS-054  guideline  should  be  carefully  planned  before  the  actual  work  starts. 
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6.2  Definition  of  the  Aims  and  Cost  Boundary 

Team’s  approach 

The  customer  for  the  results  of  the  study  is  the  Royal  Netherlands  Navy.  As  this  estimate  is  also  used  to  apply 
the  guideline  developed  by  SAS-054,  NATO  RTO  can  be  considered  as  a  (secondary)  customer.  The  aim  was 
to  practice  the  guidelines.  For  the  acquisition  cost  estimation  the  objective  was  to  compare  the  results  from  the 
SAS-076  analysis  with  the  actual  acquisition  costs. 

Lessons  Identified 

The  aim  and  the  objective  of  the  study  are  fundamental  for  the  way  the  cost  analysis  is  conducted. 

6.3  Data  and  Assumptions 

Team’s  Approach 

SAS-076  gathered  data  from  ships  similar  to  the  Netherlands  LPD’s.  The  group  applied  some  subjectivity 
in  selecting  the  ships  for  gathering  data,  but  the  group  did  not  limit  themselves  to  ships  very  close  to  the 
Netherlands’  LPDs.  Civilian  ships  like  icebreakers  were  also  included  because  data  related  to  these  ships  may 
have  information  relevant  for  the  Netherlands’  LPDs.  The  group  assumed  accuracy  in  the  data.  Engineers  or 
shipbuilders  were  used  sparingly  in  validating  the  data.  Rather,  participants  were  asked  to  validate  their  national 
data.  SAS-076  had  to  deal  with  some  limitations: 

•  No  distinction  was  possible  between  development  and  production  costs.  Not  all  data  elements  could  be 
found;  however  the  methodologies  proposed  for  acquisition  handled  missing  values. 

•  Not  all  cost  data  available  was  subdivided  into  work  breakdown  structure  items. 

Technical  data  related  to  the  ships  has  mainly  been  gathered  from  open  source.  Sources  used  were  Jane’s 
Fighting  Warships,  Federation  of  American  Scientists,  Navy  Matters,  Forecast  international,  Wikipedia,  etc. 
Regarding  cost  normalization,  the  difficulty  stands  in  the  different  currencies  (CAD,  EURO,  GBP,  USD,  NOK, 
SEK)  and  the  different  base  years  (1952  -  2010)  for  the  data  collected.  The  costs  were  normalized  to  a  common 
currency  and  then  to  a  common  base  year  (and  subsequently  adjusted  to  a  notional  common  currency  for 
sensitivity  reasons).  The  methodology  consists  of  using  an  index  to  take  into  account  inflation,  then  converting 
to  the  common  currency  with  the  exchange  rate  in  the  chosen  base  year.  Often  no  publicly  available  total  ship 
costs  index  per  nation  exist,  let  alone  a  multinational  one.  SAS-076  considered  a  multitude  of  indices  and 
approaches,  for  example: 

•  The  application  the  U.S.  Bureau  of  Fabor  Statistics  index  for  shipbuidling  labour; 

•  The  application  of  Eurostat  labour  indices; 

•  The  ratio  of  a  country’s  Ground  Domestic  Product  (GDP)  (between  original  cost  data  year  and  a  common 
year)  to  obtain  the  Then  Year  local  currency  cost  and  then  use  the  market  exchange  rate  to  get  cost  in  the 
common  currency  —  abandoned  because  it  was  deemed  too  astray  from  the  shipbuilding  industry;  and, 

•  Consumer  Price  Index  taken  from  International  Monetary  Fund  (IMF)  International  Financial  Statistics 
(IFS)  Yearbooks  and  IMF/World  Economic  Outlook  database — abandoned  because  too  far  from  ship¬ 
building  industry. 

The  final  approach  decided  upon  is  presented  in  Section  2.2. 
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Lessons  Identified 

•  It  took  quite  some  time  to  gather  all  data  required  to  estimate  the  costs  for  the  acquisition  stage.  Data 
collection  was  a  tedious  process  spanning  many  iterations  over  many  months.  Internet  searches  of  pub¬ 
licly  available  databases  were  used  to  select  the  NATO  ships  that  are  similar  in  nature  to  the  Netherlands’ 
LPDs.  Janes  Fighting  Warships  and  Federation  of  American  Scientists  were  used  to  compose  a  database 
template  of  attributes.  This  template  was  sent  around  to  SAS-076  participants  to  fill  or  validate.  Cost 
data  was  obtained  from  national  sources  contacted  by  respective  SAS-076  participants. 

•  The  group  decided  it  was  not  necessary  to  use  a  MDAL,  CARD,  DADD  or  equivalent  to  capture  data 
and  assumptions  in  a  stringent  way.  The  group  used  a  spreadsheet  containing  all  the  parameters  needed 
without  a  identification  of  sources  for  each  data  gathered.  This  simplified  approach  was  found  to  be 
sufficient  if  no  official  endorsement  is  needed  for  data  and  assumptions  collected. 

•  An  important  lesson  is  that  the  cost  normalization  procedure  is  very  important.  The  choice  of  a  certain 
index  will  have  a  major  impact  on  the  results.  SAS-054  report  is  not  very  explicit  in  the  approach  to 
follow  for  cost  normalization.  A  more  detailed  description  of  how  to  deal  with  cost  normalization  would 
be  helpful. 

6.4  Methods,  Models  and  Tools 

Team’s  approach 

As  recommended  in  the  SAS-054  Report,  two  independent  methods  were  used  for  estimating  acquisition  costs. 
For  the  acquisition  costs  the  analogy  method  and  the  parametric  method  were  used.  For  these  methods  two  data 
mining  techniques  were  applied,  which  is  a  new  method  for  cost  analysis  and  considered  to  be  very  useful: 

•  Parametric  method:  a  model  tree  that  is  a  combination  of  decision  trees  and  linear  regression  from  which 
a  Cost  Estimating  Relationships  (CERs)  is  obtained; 

•  Analogy  method:  cluster  analysis  that  consists  first  of  identifying  the  nearest  neighbours  to  both  Rotter¬ 
dam  and  Johan  de  Witt  ships  by  first  defining  a  distance  function  between  ships  and  use  the  distances  and 
known  costs  to  obtain  cost  estimates  for  the  Rotterdam  and  Johan  de  Witt. 

The  tools  investigated  were  mainly  commercial.  It  was  not  possible  to  acquire  one  of  the  tools,  due  to  the 
very  limited  resources  available.  Therefore  the  group  used  tools  that  are  publicly  available  (e.g.,  the  WEKA- 
tool). 

Lesson  Identified: 

•  The  data  available  determined,  in  a  very  large  extent,  the  choice  of  methods.  Finding  input  data  has 
been  the  criteria  to  make  the  process  go  forward.  It  was  a  continuous  interaction  between  choices  of 
methods  and  computer  models  and  availability  of  data.  For  acquisition  costs  the  group  assumed  that 
many  detailed  data  would  be  publicly  available;  and  the  group  assumed  that  for  sufficient  ships,  cost  data 
would  be  available.  This  allowed  the  data  mining  method  to  be  used.  The  data  mining  method  did  not 
constrain  the  selection  of  attributes.  Generally,  commercial  tools  fix  the  range  of  attributes. 

•  The  new  methodology  used,  based  on  data  mining,  is  very  valuable  as  it  requires  only  publicly  available 
technical  data  of  attributes  and  some  cost  data.  However,  this  method  is  not  mentioned  in  the  SAS-054 
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report.  On  the  other  hand  the  complexity  of  the  analogy  method  based  on  hierarchical  clustering  lead  to 
computational  difficulties  when  using  over  one  hundred  attributes.  This  issue  was  dealt  with  by  applying 
principal  component  analysis  and  reducing  the  dimensionality  of  the  input  without  significant  loss  of 
information. 

6.5  Risk  and  Uncertainty  Analysis 

Team’s  Approach 

SAS-076  considered  the  risk  and  uncertainty  analysis  task  implicitly:  the  two  cost  estimating  methods  produce 
probability  distributions  modeling  the  statistical  uncertainty  in  the  prediction — this  is  the  uncertainty  of  the 
models.  SAS-076  did  not  consider  a  full  range  of  uncertainty  on  the  input  attributes  or  on  the  indices  used, 
however  sensitivity  analysis  was  performed.  Another  issue  related  to  the  uncertainty  of  the  acquisition  costs 
concerns  the  missing  data.  However,  the  methodologies  used  to  estimate  the  acquisition  costs  were  able  to 
handle  missing  values.  Other  relevant  issues  related  to  the  uncertainty  of  the  acquisition  costs  regard  whether 
the  attributes  collected  by  SAS-076  describe  the  ships  completely.  It  is  not  clear  how  the  uncertainty  could  be 
considered  in  this  case  (e.g.,  if  the  costs  of  the  more  expensive  ships  arc  underestimated,  is  it  possible  that  an 
important  attribute  is  not  accounted  for?).  These  issues  are  not  explicitly  covered  by  the  group. 

Lessons  Identified 

•  Uncertainty  analysis  of  the  acquisition  cost  estimation  consisted  of  documenting  and  presenting  the  pre¬ 
dictive  capabilities  of  the  data  mining  methods  used:  their  accuracy  on  historical  systems  and  the  un¬ 
certainty  in  the  predicted  cost  of  the  Rotterdam  and  Johan  de  Witt  ships.  Probability  distributions  and 
cumulative  distribution  functions  were  developed  and  presented  graphically  and  in  tabular  format  to  allow 
a  decision  maker  to  select  the  desired  confidence  level  (and  associated  cost)  of  the  estimates. 

•  Further  uncertainty  analysis  could  have  helped,  but  was  not  undertaken.  This  could  have  consisted  of 
a  Monte  Carlo  simulation  to  model  different  inflation  and  exchange  rates,  and  could  substitute  missing 
attribute  values  (in  lieu  of  just  the  mean)  -  each  Monte  Carlo  iteration  would  feed  the  revised  data  into 
the  data  mining  algorithms  to  output  the  associated  cost  estimate.  This  effort  was  deemed  to  require 
substantial  programming  effort  to  automate  -  the  value  gained  was  not  clear. 

•  Using  cost  data  from  different  sources  may  be  a  risk.  In  this  study  it  was  not  certain  if  all  cost  data 
gathered  from  different  sources  have  used  the  same  cost  boundary.  E.g.,  personnel  costs  are  not  always 
considered  in  the  same  way.  This  issue  was  covered  by  assuming  that  all  sources  use  the  same  cost 
boundary. 

•  Risk  analysis,  interpreted  as  examining  the  chance  of  loss  of  injury  due  to  an  unfavorable  event,  was  not 
considered. 

6.6  Analysis  and  Presentation  of  Results 

Team’s  Approach 

SAS-076  did  not  define  a  strategy  for  analysis  as  such  but  undertook  the  following  ad  hoc  analyses  for  acquisi¬ 
tion  costs: 
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•  One  data  outlier  was  removed  from  the  analysis.  This  was  the  San  Antonio  LPD-17  whose  cost  overruns 
arc  well  documented.  The  LPD-17  cost  was  more  than  three  orders  of  magnitude  larger  than  the  minimum 
ship  costs. 

•  The  analysis  of  the  initial  cluster  analysis  results,  where  attributes  were  given  equal  weighting,  showed 
poor  predictive  capability  (low  correlation  of  determination,  A'2).  The  group  then  decided  to  determine 
the  set  of  attribute  weights  that  optimized  prediction  of  known  instances.  This  was  a  non-linear  convex 
optimization  problem  too  large  to  solve  in  a  reasonable  amount  of  time  and  memory.  To  circumvent 
computation  issues  subjectively  chosen  attribute  subsets  (less  than  15  at  a  time)  were  tested.  Each  of 
these  provided  a  better  R2.  To  eliminate  the  subjectivity,  a  subsequent  iteration  used  principal  component 
analysis  to  reduce  the  dimensionality  of  the  data  set  while  accounting  for  a  large  percentage  of  the  original 
data’s  variability. 

•  The  reason  for  selecting  the  M5  model  estimate  to  be  the  primary  estimate  was  due  to  the  model  con¬ 
struction:  M5  Model  trees  are  built  to  optimize  prediction  of  unseen  cases;  the  hierarchical  clustering 
approach  is  optimized  on  learning  the  known  cases. 

•  From  this  choice,  a  closer  look  at  the  CER  showed  that  rank  in  class  is  one  of  the  cost-driving  attributes. 
However,  SAS-076  had  a  doubt  regarding  Johann  de  Witt’s  rank  in  class.  Through  discussions  with 
RNLN  and  the  analysis  of  technical  data,  SAS-076  concluded  that  Johann  de  Witt  is  considered  a  separate 
class.  In  retrospect,  this  decision  lead  to  a  more  accurate  estimate. 

•  SAS-076  tried  to  determine  ANEP-41  ESWBS  cost  category  estimates.  Given  that  only  10  data  points 
(all  from  U.S.)  were  available,  the  effectiveness  of  data  mining  techniques  was  limited.  It  was  suggested 
to  take  average  percentages.  However,  results  triggered  the  team  that  ESWBS  items  were  not  interpreted 
in  the  same  way.  Also  for  not  many  ships  this  detailed  data  was  available  and  forced  the  team  to  skip  the 
method  to  estimate  costs  at  the  ESWBS  item  level. 

•  The  results  of  the  acquisition  costs  estimate  were  presented  to  the  RNLN,  using  the  probability  distribu¬ 
tion  with  uncertainties.  It  was  felt  that  the  presentation  fitted  the  customer’s  expectations. 

Lessons  Identified 

•  A  critical  approach  on  the  first  results  obtained  is  highly  recommended.  The  group  demonstrated  that 
without  this  critical  look,  the  results  obtained  would  have  been  less  satisfactory  for  the  acquisition  costs. 

7.0  CONCLUSION 

This  report  describes  two  novel  approaches  to  cost  estimation  using  known  data  mining  algorithms.  As  a  proof 
of  concept,  the  approaches  were  applied  in  a  blind  ex  post  cost  estimation  exercise  of  the  Netherlands’  landing 
platform  dock  ships. 

Both  methods  incorporate  a  multitude  of  cost  driving  factors  that  required  the  compilation  of  a  multinational 
data  set  of  dozens  of  somewhat  similar  ships.  The  data  mining  approaches  allow  for  a  greater  variability  in  the 
input  data  set — variability  that  could  be  questioned  when  using  traditional  approaches.  As  with  other  parametric 
and  analogy  approaches,  the  fidelity  of  the  estimation  models  are  very  dependent  the  data  set,  especially  if 
the  size  of  the  data  set  is  small.  Both  are  “top  down”  approaches  applicable  in  early  design  phases  of  the 
procurement  cycle. 
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The  parametric  approach  combined  features  of  decision  trees  with  linear  regression  models  to  both  classify 
similar  ships  (based  on  attributes)  and  build  piece-wise  multivariate  linear  regression  models.  The  attributes 
of  HMS  Rotterdam  class  ships  were  use  to  trace  down  the  tree  and  as  input  to  the  resulting  regression  models 
which  outputted  a  prediction. 

As  an  analogy  costing  approach,  hierarchical  agglomerative  cluster  analysis,  principal  component  analy¬ 
sis,  and  non-linear  optimization  was  used  to  calculate  a  matrix  of  distances  among  the  data  set  ships.  These 
distances  were  then  used  to  predict  the  cost  of  HMS  Rotterdam  class  ships. 

Despite  a  limited  data  set,  the  proof  of  concept  results  provide  evidence  that  the  methods  can  provide 
accurate  estimates.  The  methods  should  be  considered  for  generating  cost  estimates  for  other  systems  for 
which  ample  data  is  available. 

The  methods  herein  used  for  cost  estimating  are  new  in  the  cost  environment.  Newly  developed  methods 
in  other  areas  than  cost  analysis  should  always  be  considered  for  application  in  the  cost  analysis  area.  The  cost 
analysis  societies  should  be  aware  of  new  developments  in  methods  and  techniques  in  other  related  areas  in 
order  to  apply  them  to  cost  analysis.  This  is  in-line  with  SAS-054  recommendation  0.9.7  stating  “Research 
should  be  conducted  continuously  to  enhance  methods  and  models  for  life  cycle  costing.”  The  methods  proved 
to  be  valuable  as  they  require  only  publicly  available  technical  data  of  attributes  and  some  cost  data.  In  general, 
no  major  amendments  or  changes  to  the  SAS-054  guideline  are  required,  however  some  enhancements  are 
suggested: 

•  Differentiate  between  approaches  and  methods:  parametric,  analogy,  and  engineering  methods  can  each 
be  applied  in  a  top-down  or  bottom-up  approach. 

•  The  SAS-054  guidelines  for  cost  normalization  should  be  expanded  to  consider  multi-national  data  sets 
where  different  currencies  and  base  years  are  present.  The  approach  taken  by  SAS-076  in  this  study  can 
be  used  as  an  example. 
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Appendix  Bl-1:  DATA  SET 


Table  15:  Technical  data  for  the  United  Kingdom,  Swedish,  Norwegian,  and  French  ships. 


Data  Category  &  Element 

Albion 

Largs  Bay 

Ocean 

Carlskrona 

Atle 

Oden 

Svalbard 

Siroco 

Mistral 

I.  DESCRIPTION 

e.  Rank  in  class 

1 

1 

1 

1 

1 

1 

1 

2 

1 

f.  Vessel  type 

Mil 

Mil 

Mil 

Mil 

Civ 

Civ 

Civ 

Mil 

Mil 

II.  CONSTRUCTION 

h.  Civilian  classification  stnds.? 

N 

? 

Y 

Y 

? 

Y 

Y 

N 

Y 

III.  DIMENSIONS 

a.  Length  (m) 

176 

176.6 

203.4 

105.7 

104.6 

107.8 

103.7 

168 

199 

b.  Beam  (m) 

28.9 

26.4 

34.4 

15.2 

23.8 

31 

19.1 

23.5 

32 

c.  Draught  (m) 

7.1 

5.8 

6.6 

4 

8.3 

8.5 

6.5 

5.2 

6.2 

d.  Displacement  Light 

14600 

13690 

? 

3150 

? 

11000 

? 

8230 

16529 

e.  Displacement  Full  Load 

18500 

16160 

21758 

3800 

9500 

13000 

6300 

12400 

21600 

IV.  PERFORMANCE 

a.  Top  speed 

18 

18 

19 

20 

19 

16 

17 

21 

19 

i.  Range:  Total  distance 

8000 

8000 

8000 

? 

? 

30000 

10000 

11000 

11000 

ii.  Range:  Econ.  speed 

15 

15 

15 

? 

? 

13 

13 

15 

15 

iii.  Range:  Sailing  time 

533 

533 

533 

? 

? 

2308 

769 

733 

733 

b.  Endurance  (days) 

? 

? 

? 

? 

56 

100 

? 

30 

45 

c.  Crew:  complement 

325 

60 

491 

170 

20 

15 

48 

218 

177 

i.  Officers 

? 

? 

? 

? 

? 

? 

? 

18 

20 

ii.  Non-officers 

? 

? 

? 

? 

? 

? 

? 

200 

157 

V.  PROPULSION 

a.  Propulsion  technology 

Electric 

Electric 

Diesel 

Diesel 

Electric 

Diesel 

Electric 

Diesel 

Electric 

b.  Propeller  shafts 

2 

0 

2 

2 

4 

2 

0 

2 

0 

i.  Shaft  power  (MW) 

? 

0 

13.5 

7.76 

16.2 

18 

0 

15.3 

0 

ii.  Propeller  type 

LIPS 

N/A 

fp 

cp 

? 

cp 

N/A 

LIPS  cp 

N/A 

c.  Propulsion  pods 

0 

2 

0 

0 

? 

0 

2 

0 

2 

continued  on  next  page 
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Data  Category  &  Element 

Albion 

Largs  Bay 

Ocean 

Carlskrona 

Atle 

Oden 

Svalbard 

Siroco 

Mistral 

i.  Total  power  (MW) 

0 

4.4 

0 

0 

? 

0 

10 

0 

14 

d.  Net  power  (MW) 

? 

? 

13.5 

7.76 

? 

18 

10 

15.3 

14 

e.  Bow  Thrusters 

1 

1 

1 

? 

? 

0 

1 

1 

1 

i.  Total  power  (MW) 

0.865 

? 

0.45 

? 

? 

0 

? 

0.735 

1.5 

VI.  ELECT  POWER  GEN. 

a.  Generators 

4 

4 

? 

? 

? 

4 

4 

5 

4 

i.  Total  capacity  (MW) 

15.6 

11.2 

? 

? 

16.2 

? 

13.56 

4.25 

20.8 

ii.  Generator  technology 

Diesel 

Diesel 

Diesel 

? 

Diesel 

Diesel 

Diesel 

Diesel 

Diesel 

VII.  LIFT  CAPACITY 

a.  Vehicle  fuel  (litres) 

? 

? 

? 

? 

? 

? 

? 

? 

? 

b.  Aviation  fuel  (litres) 

? 

? 

? 

? 

? 

? 

? 

? 

1071.3 

c.  Fresh  water  (litres) 

? 

? 

? 

? 

? 

310000 

? 

? 

380 

d.  Bulk  cargo  space  (m3) 

? 

? 

? 

? 

? 

? 

? 

? 

? 

e.  Vehicle  space  (m2) 

? 

2875 

? 

0 

0 

0 

0 

? 

2650 

f.  Well  deck  (Y/N) 

Y 

Y 

N 

N 

N 

N 

N 

Y 

Y 

i.  Length  (m) 

? 

95.5 

0 

0 

0 

0 

0 

122 

? 

ii.  Width  (m) 

? 

14 

0 

0 

0 

0 

0 

14.2 

? 

iii.  Capacity  ( m 2) 

? 

1440 

0 

0 

0 

0 

0 

1732.4 

885 

iv.  LCAC 

2 

0 

0 

0 

0 

0 

0 

? 

2 

v.  LCM  6 

? 

? 

0 

0 

0 

0 

0 

? 

? 

vi.  LCM  8 

? 

? 

0 

0 

0 

0 

0 

? 

? 

vii.  LCU 

4 

2 

0 

0 

0 

0 

0 

? 

4 

viii.  LVT 

? 

? 

0 

0 

0 

0 

0 

? 

? 

ix.  LCVP 

? 

2 

0 

0 

0 

0 

0 

? 

? 

x.  LCPL 

? 

? 

0 

0 

0 

0 

0 

? 

? 

xi.  EFV 

? 

? 

0 

0 

0 

0 

0 

? 

? 

g.  Cargo/Aircraft  Elevator/Lifts 

? 

? 

2 

0 

? 

? 

0 

1 

3 

i.  Capacity  <  5  tonnes 

? 

? 

? 

0 

? 

? 

0 

0 

? 

ii.  5  <  capacity  <10  tonnes 

? 

? 

? 

0 

? 

? 

0 

0 

? 

iii.  10  <  capacity  <  15  tonnes 

? 

? 

? 

0 

? 

? 

0 

0 

? 

continued  on  next  page 
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Data  Category  &  Element 

Albion 

Largs  Bay 

Ocean 

Carlskrona 

Atle 

Oden 

Svalbard 

Siroco 

Mistral 

iv.  Capacity  >15  tonnes 

? 

? 

? 

0 

? 

? 

0 

1 

? 

h.  Cranes 

? 

2 

? 

? 

? 

3 

0 

1 

? 

i.  Capacity  <  5  tonnes 

? 

0 

? 

? 

? 

1 

0 

0 

? 

ii.  5  <  capacity  <10  tonnes 

? 

0 

? 

? 

? 

1 

0 

0 

? 

iii.  10  <  capacity  <15  tonnes 

? 

0 

? 

? 

? 

1 

0 

0 

? 

iv.  15  <  capacity  <  20  tonnes 

? 

0 

? 

? 

? 

0 

0 

0 

? 

v.  20  <  capacity  <  25  tonnes 

? 

0 

? 

? 

? 

0 

0 

0 

? 

vi.  25  <  capacity  <  30  tonnes 

? 

2 

? 

? 

? 

0 

0 

0 

? 

vii.  30  <  capacity  <  40  tonnes 

? 

0 

? 

? 

? 

0 

0 

1 

? 

viii.  40  <  capacity  <  50  tonnes 

? 

0 

? 

? 

? 

0 

0 

0 

? 

ix.  50  <  capacity  <  60  tonnes 

? 

0 

? 

? 

? 

0 

0 

0 

? 

x.  Capacity  >  60  tonnes 

? 

0 

? 

? 

? 

0 

0 

0 

? 

i.  Berthing  (troop  capacity):  Baseline 

305 

356 

480 

? 

? 

65 

? 

470 

450 

j.  Berthing  (troop  capacity):  Surge 

305 

344 

323 

? 

? 

? 

? 

? 

450 

VIII.  FLIGHT  DECK 

a.  Equipped  with  flight  deck?  (Y/N) 

Y 

Y 

Y 

Y 

? 

? 

Y 

Y 

Y 

b.  Flight  deck  length  (m) 

64 

? 

170 

? 

? 

? 

? 

? 

199 

c.  Flight  deck  width  (m) 

? 

? 

31.7 

? 

? 

? 

? 

? 

32 

d.  Flight  deck  area  (m2) 

? 

? 

5389 

? 

? 

? 

? 

1740 

6368 

e.  Helicopter  landing  spots  (max.  #) 

2 

2 

6 

? 

? 

? 

1 

3 

6 

i.  Merlin  /  Sea  King 

2 

2 

6 

? 

? 

? 

? 

? 

? 

ii.  NH  90  /  Lynx  /  Puma  /  Cougar 

? 

? 

? 

? 

? 

? 

? 

3 

6 

iii.  CH-46E  Sea  Knight 

? 

? 

? 

? 

? 

? 

? 

? 

? 

iv.  CH-53  Sea  Stallion 

? 

? 

? 

? 

? 

? 

? 

? 

1 

v.  MV- 22  Osprey 

? 

? 

? 

? 

? 

? 

? 

? 

1 

f.  Chinook  capable  (Yes/No) 

Y 

Y 

Y 

N 

? 

? 

N 

? 

Y 

g.  Equipped  with  hangar?  (Y/N) 

N 

N 

Y 

N 

? 

? 

N 

? 

Y 

h.  Hangar  size  (m2) 

0 

0 

2338 

0 

? 

? 

0 

? 

1800 

i.  Helicopters  supported  (max.  #) 

2 

? 

18 

0 

? 

? 

1 

4 

16 

i.  Merlin  /  Sea  King 

? 

? 

12 

0 

? 

? 

? 

? 

? 
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Data  Category  &  Element 

Albion 

Largs  Bay 

Ocean 

Carlskrona 

Atle 

Oden 

Svalbard 

Siroco 

Mistral 

ii.  NH  90  /  Lynx  /  Puma  /  Cougar 

? 

? 

6 

0 

? 

? 

1 

4 

16 

iii.  CH-46E  Sea  Knight 

? 

? 

? 

0 

? 

? 

? 

? 

? 

iv.  CH-53  Sea  Stallion 

? 

? 

? 

0 

? 

? 

? 

? 

? 

v.  MV-22  Osprey 

? 

? 

? 

0 

? 

? 

? 

? 

? 

IX.  ARMAMENT 

a.  Guns  (calibre  >  75mm) 

0 

0 

0 

0 

0 

0 

0 

0 

0 

b.  Guns  (50mm  <  calibre  <  75mm) 

0 

0 

0 

2 

0 

0 

1 

0 

0 

c.  Guns  (30mm  <  calibre  <  50mm) 

0 

0 

0 

2 

0 

0 

0 

3 

2 

d.  Guns  (20mm  <  calibre  <  30mm) 

2 

0 

8 

0 

0 

0 

0 

0 

0 

e.  30mm  CIWS  (Goalkeeper) 

2 

0 

0 

0 

0 

0 

0 

0 

0 

f.  20mm  CIWS  (Phalanx) 

0 

0 

3 

0 

0 

0 

0 

0 

0 

g.  Machine  guns  (12.7mm) 

0 

0 

0 

0 

0 

0 

? 

4 

4 

h.  Machine  guns  (7.62mm) 

4 

6 

4 

0 

0 

0 

? 

0 

0 

i.  SSM  launchers 

0 

0 

0 

0 

0 

0 

0 

0 

0 

j.  SAM  launchers 

0 

0 

0 

0 

0 

0 

0 

3 

2 

k.  #  of  torpedos  earned 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1.  Torpedo  tubes 

0 

0 

0 

0 

0 

0 

0 

0 

0 

m.  Torpedo  launchers 

0 

0 

0 

0 

0 

0 

0 

0 

0 

X.  COUNTERMEASURES 

a.  Chaff  launchers 

8 

? 

8 

2 

0 

0 

0 

0 

0 

b.  Toipedo  decoys 

1 

? 

1 

0 

0 

0 

0 

1 

1 

c.  Other  systems 

0 

? 

0 

0 

0 

0 

0 

0 

0 

d.  Number  of  ESM  systems 

1 

? 

1 

1 

0 

0 

0 

0 

1 

e.  Number  of  ECM  systems 

1 

? 

1 

0 

0 

0 

0 

2 

0 

XI.  RADARS/TACAN/IFF/SONARS 

a.  Total  radar  systems  mounted 

3 

1 

4 

5 

? 

? 

2 

4 

3 

i.  A-band 

0 

0 

0 

0 

? 

? 

0 

0 

0 

ii.  B-band 

0 

0 

0 

0 

? 

? 

0 

0 

0 

iii.  C-band 

0 

0 

0 

0 

? 

? 

0 

0 

0 

iv.  D-band 

0 

0 

0 

0 

? 

? 

0 

1 

0 
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Data  Category  &  Element 

Albion 

Largs  Bay 

Ocean 

Carlskrona 

Atle 

Oden 

Svalbard 

Siroco 

Mistral 

v.  E-band 

2 

0 

2 

1 

? 

? 

0 

0 

0 

vi.  F-band 

2 

0 

2 

1 

? 

? 

0 

0 

0 

vii.  G-band 

0 

0 

0 

1 

? 

? 

1 

0 

1 

viii.  H-band 

0 

0 

0 

1 

? 

? 

0 

0 

0 

ix.  I-band 

1 

1 

2 

4 

? 

? 

1 

3 

2 

x.  J-band 

0 

0 

0 

2 

? 

? 

0 

0 

0 

b.  #  of  TACAN/IFF  systems  mounted 

1 

? 

1 

? 

? 

? 

? 

? 

? 

c.  #  of  distinct  sonar  systems  mounted 

0 

? 

0 

? 

? 

? 

0 

? 

0 

XII.  COMBAT  DATA  SYSTEMS 

a.  Number  of  distinct  systems 

5 

? 

6 

? 

? 

? 

? 

5 

7 

XIII.  WEAPONS  CONTROL  SYSTEMS 

a.  Number  of  distinct  systems 

2 

? 

? 

? 

? 

? 

? 

2 

2 

XIV.  OTHER  CAPABILITIES 

a.  Equipped  with  hospital?  (Y/N) 

? 

? 

? 

Y 

? 

Y 

N 

Y 

Y 

i.  Number  of  beds 

? 

? 

? 

? 

? 

? 

0 

47 

69 

ii.  Operating  rooms 

? 

? 

? 

? 

? 

? 

0 

2 

2 

iii.  X-Ray  facility  (Y/N) 

? 

? 

? 

? 

? 

? 

N 

? 

Y 

b.  Dental  capability  (Y/N) 

? 

? 

? 

? 

? 

? 

N 

? 

? 

c.  Command/Control  facility  (Y/N) 

Y 

? 

Y 

? 

? 

? 

N 

? 

Y 

d.  NBCD  Facilities  (Y/N) 

? 

Y 

Y 

? 

? 

? 

Y 

? 

? 
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Table  16:  Technical  data  for  the  United  States  and  Canadian  ships. 


Data  Category  &  Element 

Thomaston 

Anchorage 

Whidbey  Island 

Harpers  Ferry 

Raleigh 

Austin 

San  Antonio 

Protecteur 

I.  DESCRIPTION 

e.  Rank  in  class 

1 

1 

1 

1 

1 

1 

1 

1 

f.  Vessel  type 

Mil 

Mil 

Mil 

Mil 

Mil 

Mil 

Mil 

Mil 

II.  CONSTRUCTION 

h.  Civilian  classification  stnds.? 

N 

N 

N 

N 

N 

N 

N 

N 

III.  DIMENSIONS 

a.  Length  (m) 

160 

168.6 

185.8 

185.8 

159.1056 

173.8 

208.4 

171.9 

b.  Beam  (m) 

26 

25.6 

25.6 

25.6 

30.48 

30.5 

31.9 

23.2 

c.  Draught  (m) 

5.8 

6 

6.3 

6.3 

7.0104 

7 

7 

10.46 

d.  Displacement  Light 

9042 

8600 

11125 

11125 

8789 

9130 

18477 

9259 

e.  Displacement  Full  Load 

11710 

13700 

15939 

16740 

14339 

17244 

25885 

25676 

IV.  PERFORMANCE 

a.  Top  speed 

21 

22 

22 

22 

21 

21 

22 

21 

i.  Range:  Total  distance 

13000 

14800 

8000 

8000 

9600 

7700 

? 

7500 

ii.  Range:  Econ.  speed 

10 

12 

18 

18 

16 

20 

? 

11.5 

iii.  Range:  Sailing  time 

? 

1233 

444 

444 

600 

385 

? 

652 

b.  Endurance  (days) 

? 

? 

? 

? 

? 

60 

? 

? 

c.  Crew:  complement 

341 

374 

413 

352 

429 

420 

396 

335 

i.  Officers 

29 

24 

21 

24 

29 

24 

32 

38 

ii.  Non-officers 

312 

350 

392 

328 

400 

396 

364 

297 

V.  PROPULSION 

a.  Propulsion  technology 

Steam 

Steam 

Diesel 

Diesel 

Steam 

Steam 

Diesel 

Steam 

b.  Propeller  shafts 

2 

2 

2 

2 

2 

2 

2 

1 

i.  Shaft  power  (MW) 

17 

17.9 

24.6 

24.6 

17.9 

17.9 

29.84 

15.7 

ii.  Propeller  type 

? 

? 

cp 

cp 

? 

? 

cp 

? 

c.  Propulsion  pods 

0 

0 

0 

0 

0 

0 

0 

0 

i.  Total  power  (MW) 

0 

0 

0 

0 

0 

0 

0 

0 

d.  Net  power  (MW) 

17 

17.9 

24.6 

24.6 

17.9 

17.9 

29.84 

15.7 

e.  Bow  Thrusters 

0 

0 

0 

0 

0 

0 

0 

1 
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Data  Category  &  Element 

Thomaston 

Anchorage 

Whidbey  Island 

Harpers  Ferry 

Raleigh 

Austin 

San  Antonio 

Protecteur 

i.  Total  power  (MW) 

0 

0 

0 

0 

0 

0 

0 

? 

VI.  ELECT.  POWER  GEN. 

a.  Generators 

? 

? 

4 

4 

? 

4 

5 

? 

i.  Total  capacity  (MW) 

? 

? 

5.2 

5.2 

? 

? 

12.5 

? 

ii.  Generator  technology 

Steam 

Steam 

Diesel 

Diesel 

? 

Steam 

Diesel 

? 

VII.  LIFT  CAPACITY 

a.  Vehicle  fuel  (litres) 

? 

? 

? 

? 

? 

? 

37850 

? 

b.  Aviation  fuel  (litres) 

? 

110200 

110000 

110000 

? 

? 

1173500 

? 

c.  Fresh  water  (litres) 

? 

? 

? 

? 

? 

? 

? 

? 

d.  Bulk  cargo  space  (m3) 

99 

? 

141.6 

1914 

? 

? 

963 

? 

e.  Vehicle  space  (m2) 

? 

? 

1161 

1877 

? 

? 

2285 

? 

f.  Well  deck  (Y/N) 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

N 

i.  Length  (m) 

120.7 

131.1 

134.1 

? 

51.2 

120.1 

? 

0 

ii.  Width  (m) 

15.24 

15.2 

15.2 

15.2 

15.2 

15.2 

? 

0 

iii.  Capacity  (in2) 

1840 

1992.72 

2038.32 

? 

778.24 

1825.52 

? 

0 

iv.  LCAC 

? 

3 

4 

2 

2 

2 

2 

0 

v.  LCM  6 

21 

18 

21 

9 

3 

9 

? 

0 

vi.  LCM  8 

9 

9 

10 

4 

4 

4 

? 

0 

vii.  LCU 

3 

3 

3 

1 

1 

? 

1 

0 

viii.  LVT 

48 

50 

64 

64 

20 

20 

? 

0 

ix.  LCVP 

? 

? 

? 

? 

? 

4 

? 

4 

x.  LCPL 

? 

? 

2 

2 

? 

4 

? 

0 

xi.  EFV 

? 

? 

? 

? 

? 

? 

14 

0 

g.  Cargo/Aircraft  Elevator/Lifts 

? 

? 

0 

0 

? 

1 

3 

0 

i.  Capacity  <  5  tonnes 

? 

? 

0 

0 

? 

0 

1 

0 

ii.  5  <  capacity  <10  tonnes 

? 

? 

0 

0 

? 

1 

2 

0 

iii.  10  <  capacity  <  15  tonnes 

? 

? 

0 

0 

? 

0 

0 

0 

iv.  Capacity  >15  tonnes 

? 

? 

0 

0 

? 

0 

0 

0 

h.  Cranes 

3 

2 

3 

1 

? 

7 

2 

2 

i.  Capacity  <  5  tonnes 

0 

0 

0 

0 

? 

6 

2 

0 
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Data  Category  &  Element 

Thomas  to  n 

Anchorage 

Whidbey  Island 

Harpers  Ferry 

Raleigh 

Austin 

San  Antonio 

Protecteur 

ii.  5  <  capacity  <10  tonnes 

1 

0 

0 

0 

? 

0 

0 

0 

iii.  10  <  capacity  <  15  tonnes 

0 

0 

1 

0 

? 

0 

0 

2 

iv.  15  <  capacity  <  20  tonnes 

0 

0 

1 

0 

? 

0 

0 

0 

v.  20  <  capacity  <  25  tonnes 

0 

0 

0 

0 

? 

0 

0 

0 

vi.  25  <  capacity  <  30  tonnes 

0 

0 

0 

1 

? 

1 

0 

0 

vii.  30  <  capacity  <  40  tonnes 

0 

0 

0 

0 

? 

0 

0 

0 

viii.  40  <  capacity  <  50  tonnes 

2 

2 

0 

0 

? 

0 

0 

0 

ix.  50  <  capacity  <  60  tonnes 

0 

0 

1 

0 

? 

0 

0 

0 

x.  Capacity  >  60  tonnes 

0 

0 

0 

0 

? 

0 

0 

0 

i.  Berthing  (troop  capacity):  Baseline 

325 

366 

402 

402 

930 

900 

699 

0 

j.  Berthing  (troop  capacity):  Surge 

? 

? 

102 

102 

? 

? 

101 

0 

VIII.  FLIGHT  DECK 

a.  Equipped  with  flight  deck?  (Y/N) 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

b.  Flight  deck  length  (m) 

? 

? 

64.6 

? 

? 

51.2 

? 

? 

c.  Flight  deck  width  (m) 

? 

? 

25.3 

? 

? 

? 

? 

? 

d.  Flight  deck  area  (m2) 

? 

? 

1634.38 

? 

? 

? 

? 

? 

e.  Helicopter  landing  spots  (max.  #) 

1 

1 

2 

2 

? 

2 

4 

? 

i.  Merlin  /  Sea  King 

? 

? 

? 

? 

? 

? 

? 

? 

ii.  NH  90  /  Lynx  /  Puma  /  Cougar 

? 

? 

? 

? 

? 

? 

? 

? 

iii.  CH-46E  Sea  Knight 

? 

? 

? 

? 

? 

2 

4 

? 

iv.  CH-53  Sea  Stallion 

? 

1 

2 

2 

? 

2 

2 

? 

v.  MV-22  Osprey 

? 

? 

? 

? 

? 

0 

2 

0 

f.  Chinook  capable  (Yes/No) 

? 

? 

? 

? 

? 

? 

? 

N 

g.  Equipped  with  hangar?  (Y/N) 

? 

N 

N 

N 

N 

N 

Y 

Y 

h.  Hangar  size  (m2) 

? 

0 

0 

0 

0 

0 

? 

? 

i.  Helicopters  supported  (max.  #) 

8 

1 

2 

0 

6 

6 

2 

3 

i.  Merlin  /  Sea  King 

? 

? 

0 

0 

? 

? 

? 

3 

ii.  NH  90  /  Lynx  /  Puma  /  Cougar 

? 

? 

0 

0 

? 

? 

? 

0 

iii.  CH-46E  Sea  Knight 

? 

? 

0 

0 

6 

6 

2 

0 

iv.  CH-53  Sea  Stallion 

? 

1 

2 

0 

? 

3 

1 

0 
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Data  Category  &  Element 

Thomaston 

Anchorage 

Whidbey  Island 

Harpers  Ferry 

Raleigh 

Austin 

San  Antonio 

Protecteur 

v.  MV-22  Osprey 

? 

? 

0 

0 

? 

? 

1 

0 

IX.  ARMAMENT 

a.  Guns  (calibre  >  75mm) 

4 

4 

0 

0 

8 

4 

0 

1 

b.  Guns  (50mm  <  calibre  <  75mm) 

0 

0 

0 

0 

0 

0 

0 

0 

c.  Guns  (30mm  <  calibre  <  50mm) 

0 

0 

0 

0 

0 

0 

2 

0 

d.  Guns  (20mm  <  calibre  <  30mm) 

6 

0 

2 

2 

0 

0 

0 

0 

e.  30mm  CIWS  (Goalkeeper) 

0 

0 

0 

0 

0 

0 

0 

0 

f.  20mm  CIWS  (Phalanx) 

0 

0 

2 

2 

0 

0 

0 

0 

g.  Machine  guns  (12.7mm) 

? 

6 

6 

6 

0 

8 

4 

6 

h.  Machine  guns  (7.62mm) 

? 

0 

0 

0 

0 

0 

0 

0 

i.  SSM  launchers 

? 

0 

0 

0 

0 

0 

0 

? 

j.  SAM  launchers 

? 

0 

2 

2 

0 

0 

2 

? 

k.  #  of  toipedos  earned 

? 

0 

0 

0 

0 

0 

0 

0 

1.  Torpedo  tubes 

? 

0 

0 

0 

0 

0 

0 

0 

m.  Torpedo  launchers 

? 

0 

0 

0 

0 

0 

0 

0 

X.  COUNTERMEASURES 

a.  Chaff  launchers 

? 

0 

4 

6 

0 

0 

6 

6 

b.  Torpedo  decoys 

? 

0 

1 

1 

0 

0 

1 

0 

c.  Other  systems 

? 

0 

0 

0 

0 

0 

0 

0 

d.  Number  of  ESM  systems 

? 

1 

2 

2 

1 

1 

1 

1 

e.  Number  of  ECM  systems 

? 

0 

0 

0 

0 

0 

1 

0 

XI.  RADARS/TACAN/IFF/SONARS 

a.  Total  radar  systems  mounted 

? 

3 

3 

3 

3 

3 

3 

3 

i.  A-band 

? 

0 

0 

0 

0 

0 

0 

0 

ii.  B-band 

? 

0 

0 

0 

0 

1 

0 

0 

iii.  C-band 

? 

0 

1 

1 

0 

0 

0 

0 

iv.  D-band 

? 

0 

0 

0 

0 

0 

0 

0 

v.  E-band 

? 

1 

0 

0 

1 

0 

1 

0 

vi.  F-band 

? 

1 

0 

0 

1 

0 

1 

0 

vii.  G-band 

? 

1 

1 

1 

1 

1 

0 

1 

continued  on  next  page 
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ORGANIZATION 


continued  from  previous  page 


Data  Category  &  Element 

Thomaston 

Anchorage 

Whidbey  Island 

Harpers  Ferry 

Raleigh 

Austin 

San  Antonio 

Protecteur 

viii.  H-band 

? 

0 

0 

0 

0 

0 

0 

0 

ix.  I-band 

? 

1 

1 

1 

1 

1 

2 

2 

x.  J-band 

? 

0 

1 

1 

0 

0 

0 

0 

b.  #  of  TACAN/IFF  systems  mounted 

? 

? 

1 

1 

1 

1 

? 

1 

c.  #  of  distinct  sonar  systems  mounted 

? 

0 

0 

0 

0 

0 

0 

0 

XII.  COMBAT  DATA  SYSTEMS 

a.  Number  of  distinct  systems 

? 

2 

3 

3 

2 

3 

5 

2 

XIII.  WEAPONS  CONTROL  SYSTEMS 

a.  Number  of  distinct  systems 

? 

? 

? 

? 

? 

? 

? 

? 

XIV.  OTHER  CAPABILITIES 

a.  Equipped  with  hospital?  (Y/N) 

N 

? 

Y 

Y 

? 

Y 

Y 

Y 

i.  Number  of  beds 

0 

? 

? 

? 

? 

12 

24 

8 

ii.  Operating  rooms 

0 

? 

? 

? 

? 

? 

2 

1 

iii.  X-Ray  facility  (Y/N) 

N 

? 

? 

? 

? 

Y 

? 

Y 

b.  Dental  capability  (Y/N) 

N 

? 

Y 

Y 

? 

Y 

Y 

Y 

c.  Command/Control  facility  (Y/N) 

N 

? 

? 

? 

? 

Y 

? 

? 

d.  NBCD  Facilities  (Y/N) 

? 

? 

? 

? 

? 

? 

? 

Y 
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Appendix  B 1-2:  PRINCIPAL  COMPONENT  DATA  SET 


Table  17:  Principal  component  analysis  (80%  coverage):  resulting  attributes. 


Name  At  A2  A3  A4  A5  A6  A7  A8  A9  A10 

Rotterdam 


Johan  de  Witt 

Thomaston 

Plymouth  Rock 

Fort  Snelling 

Point  Defiance 

Spiegel  Grove 

Alamo 

Hermitage 

Monticello 

Anchorage 

Portland 

Pensacola 

Mount  Vernon 

Fort  Fisher 

Whidbey  Island 

Germantown 

Fort  McHenry 

Gunston  Hall 

Comstock 

Tortuga 

Rushmore 

Ashland 

Harpers  Ferry 

Carter  Hall 

Oak  Hill 

Pearl  Harbour 

Raleigh 

Vancouver 

La  Salle 

Austin 

Ogden 

Duluth 

Cleveland 

Dubuque 

Denver 

Juneau 

Coronado 

Shreveport 

Nashville 

Trenton 

Ponce 

Svalbard 

Carlskrona 

Atle 

Oden 

Protecteur 

Preserver 

Albion 

Bulwark 

Largs  Bay 

Lyme  Bay 

Mounts  Bay 

Cardigan  Bay 

Ocean 

Siroco 

Mistral 

Tonnerre 

Dixmude  (BPC3) 


-0.6206 

2.4141 

-1.2649 

2.8019 

1.9276 

3.1298 

1.9379 

3.1359 

1.9482 

3.1419 

1.9585 

3.1480 

1.9687 

3.1541 

1.9790 

3.1602 

1.9893 

3.1662 

1.9995 

3.1723 

1.7511 

3.4262 

1.7614 

3.4323 

1.7716 

3.4384 

1.7819 

3.4444 

1.7922 

3.4505 

2.2519 

3.5869 

2.2622 

3.5930 

2.2724 

3.5991 

2.2827 

3.6052 

2.2930 

3.6112 

2.3032 

3.6173 

2.3135 

3.6234 

2.3238 

3.6294 

1.6647 

3.2549 

1.7338 

3.3117 

1.7440 

3.3178 

1.7543 

3.3239 

1.3761 

3.3173 

1.3864 

3.3233 

1.3967 

3.3294 

2.1235 

4.0747 

2.1338 

4.0808 

2.1441 

4.0868 

2.1509 

4.0852 

2.1612 

4.0913 

2.1714 

4.0973 

2.1817 

4.1034 

2.1920 

4.1095 

2.2022 

4.1155 

2.2125 

4.1216 

2.2262 

4.1354 

2.2365 

4.1415 

-0.7433 

1.1388 

0.0059 

0.9827 

0.0467 

1.9025 

0.0028 

1.5587 

0.4067 

1.8877 

0.4170 

1.8938 

-0.0074 

3.5859 

0.0029 

3.5920 

-0.3477 

3.5454 

-0.3375 

3.5515 

-0.3272 

3.5575 

-0.3169 

3.5636 

-1.3561 

3.0070 

-0.0044 

3.5500 

-2.0123 

5.0343 

-2.0020 

5.0404 

-1.9670 

5.0010 

0.7228 

-0.9292 

0.5701 

-0.5635 

-0.1413 

1.5096 

-0.1478 

1.5097 

-0.1543 

1.5099 

-0.1608 

1.5101 

-0.1673 

1.5103 

-0.1738 

1.5104 

-0.1803 

1.5106 

-0.1868 

1.5108 

0.0422 

0.1333 

0.0357 

0.1335 

0.0292 

0.1337 

0.0227 

0.1339 

0.0162 

0.1340 

2.5713 

-0.4678 

2.5648 

-0.4676 

2.5583 

-0.4674 

2.5518 

-0.4673 

2.5453 

-0.4671 

2.5388 

-0.4669 

2.5323 

-0.4667 

2.5258 

-0.4666 

2.1292 

-0.8805 

2.1975 

-0.7664 

2.1910 

-0.7663 

2.1845 

-0.7661 

-0.3228 

-1.0441 

-0.3293 

-1.0439 

-0.3358 

-1.0437 

-1.0059 

-1.4247 

-1.0124 

-1.4245 

-1.0189 

-1.4243 

-1.0177 

-1.4156 

-1.0242 

-1.4154 

-1.0307 

-1.4153 

-1.0372 

-1.4151 

-1.0437 

-1.4149 

-1.0502 

-1.4147 

-1.0567 

-1.4146 

-1.0709 

-1.4229 

-1.0774 

-1.4227 

-0.4784 

-0.0210 

0.8272 

-1.3437 

-0.0422 

-1.3997 

0.0901 

-1.2462 

0.6018 

-1.6665 

0.5953 

-1.6663 

0.8295 

-0.4372 

0.8230 

-0.4370 

0.2793 

-0.3506 

0.2728 

-0.3504 

0.2663 

-0.3502 

0.2598 

-0.3500 

0.8555 

-1.2696 

1.6121 

-0.4833 

0.5142 

0.1228 

0.5077 

0.1230 

0.7382 

0.2820 

-0.4955 

0.7275 

-0.8626 

0.4325 

0.7688 

-0.1155 

0.7631 

-0.1156 

0.7574 

-0.1156 

0.7517 

-0.1157 

0.7460 

-0.1158 

0.7403 

-0.1159 

0.7346 

-0.1160 

0.7290 

-0.1161 

0.1469 

0.1039 

0.1412 

0.1038 

0.1355 

0.1038 

0.1298 

0.1037 

0.1241 

0.1036 

-0.9544 

-0.4128 

-0.9600 

-0.4129 

-0.9657 

-0.4130 

-0.9714 

-0.4131 

-0.9771 

-0.4132 

-0.9828 

-0.4133 

-0.9885 

-0.4134 

-0.9942 

-0.4134 

-0.5622 

-0.2858 

-0.5860 

-0.2634 

-0.5917 

-0.2635 

-0.5973 

-0.2636 

0.0914 

0.1091 

0.0857 

0.1090 

0.0800 

0.1089 

-0.6041 

0.2968 

-0.6098 

0.2967 

-0.6155 

0.2966 

-0.6195 

0.2900 

-0.6252 

0.2899 

-0.6308 

0.2899 

-0.6365 

0.2898 

-0.6422 

0.2897 

-0.6479 

0.2896 

-0.6536 

0.2895 

-0.6610 

0.2959 

-0.6667 

0.2958 

-0.1696 

-0.7977 

0.1930 

-0.0546 

0.1531 

-0.3840 

0.6961 

-0.7923 

0.3229 

-1.0168 

0.3172 

-1.0168 

-0.1269 

0.0663 

-0.1326 

0.0662 

-1.0843 

0.0178 

-1.0900 

0.0177 

-1.0957 

0.0177 

-1.1013 

0.0176 

0.6200 

-0.7988 

-0.1249 

2.0335 

-1.1827 

-0.4498 

-1.1884 

-0.4499 

-1.4906 

-0.5423 

2.2589 

1.8520 

2.3956 

1.3758 

0.4850 

1.1027 

0.4846 

1.1057 

0.4841 

1.1087 

0.4836 

1.1117 

0.4831 

1.1147 

0.4826 

1.1176 

0.4821 

1.1206 

0.4816 

1.1236 

0.4367 

1.1533 

0.4362 

1.1562 

0.4357 

1.1592 

0.4352 

1.1622 

0.4347 

1.1652 

0.0767 

1.0708 

0.0763 

1.0738 

0.0758 

1.0768 

0.0753 

1.0798 

0.0748 

1 .0828 

0.0743 

1.0858 

0.0738 

1.0887 

0.0733 

1.0917 

0.1686 

0.7128 

0.2029 

0.7011 

0.2024 

0.7041 

0.2019 

0.7071 

0.0184 

0.8934 

0.0179 

0.8964 

0.0174 

0.8994 

0.0522 

1.0929 

0.0518 

1.0959 

0.0513 

1.0988 

0.0523 

1.0992 

0.0518 

1.1022 

0.0514 

1.1052 

0.0509 

1.1082 

0.0504 

1.1112 

0.0499 

1.1141 

0.0494 

1.1171 

0.0474 

1.1227 

0.0469 

1.1257 

-0.3938 

-0.2546 

-1.1465 

1.1058 

-0.7034 

0.2808 

-1.0531 

0.4929 

-0.0801 

1.1704 

-0.0806 

1.1734 

1.7115 

0.3668 

1.7110 

0.3698 

0.7886 

-1.0940 

0.7881 

-1.0910 

0.7877 

-1.0880 

0.7872 

-1.0850 

0.0159 

2.1329 

-0.2927 

1.1556 

-0.2610 

1.3142 

-0.2615 

1.3172 

-0.3208 

1.4085 

-0.3011 

1.2791 

0.0415 

1.4135 

-0.3971 

0.5587 

-0.3994 

0.5546 

-0.4017 

0.5505 

-0.4040 

0.5463 

-0.4063 

0.5422 

-0.4086 

0.5381 

-0.4109 

0.5340 

-0.4132 

0.5298 

0.5488 

1.2309 

0.5465 

1.2268 

0.5442 

1.2227 

0.5419 

1.2185 

0.5396 

1.2144 

-0.3635 

1.0883 

-0.3658 

1.0842 

-0.3681 

1.0801 

-0.3704 

1.0759 

-0.3727 

1.0718 

-0.3750 

1.0677 

-0.3773 

1.0635 

-0.3796 

1.0594 

-0.4586 

0.9417 

-0.4489 

0.9213 

-0.4512 

0.9172 

-0.4535 

0.9131 

-0.3140 

1.3569 

-0.3163 

1.3528 

-0.3186 

1.3487 

-0.2233 

0.9882 

-0.2256 

0.9841 

-0.2279 

0.9800 

-0.2269 

0.9703 

-0.2292 

0.9662 

-0.2315 

0.9621 

-0.2338 

0.9579 

-0.2361 

0.9538 

-0.2384 

0.9497 

-0.2407 

0.9456 

-0.2464 

0.9470 

-0.2487 

0.9428 

-0.0443 

1.0611 

-0.1787 

2.0847 

-0.2888 

1.5649 

-0.0974 

1.4152 

0.0202 

0.5590 

0.0179 

0.5549 

-0.8352 

2.2346 

-0.8376 

2.2304 

-0.5283 

0.9502 

-0.5306 

0.9461 

-0.5329 

0.9420 

-0.5352 

0.9378 

-2.6844 

1.2224 

0.5445 

1.1961 

0.2379 

0.5384 

0.2355 

0.5343 

0.2621 

0.7027 

RTO-TR-SAS-076 


B1  -53 


ANNEX  B1  -  ESTIMATING  THE  ACQUISITION 
COST  OF  THE  ROYAL  NETHERLANDS  NAVY 
LANDING  PLATFORM  DOCK  SHIPS:  AN  EX  POST  ANALYSIS 


ORGANIZATION 


Appendix  B 1-3:  M5  MODEL  TREE  ALGORITHM 

This  Annex  details  the  M5  model  tree  algorithm.  It  does  not  present  original  research  and  the  algorithm  itself 
should  not  be  attributed  to  the  authors  of  this  paper.  The  reader  is  referred  to  Witten  and  Frank  [14]  and  Wang 
and  Witten  [ 1 5  ]  for  further  details. 

The  description  of  the  algorithm  is  divided  into  four  subsections:  building  the  decision  tree,  fitting  linear 
regression  models,  pruning  the  tree,  and  finally  smoothing  the  tree.  The  two  final  subsections  of  this  annex 
detail  how  the  M5  system  handles  nominal  attributes  and  missing  data. 

Building  the  Initial  Decision  Tree 

The  M5  model  tree  algorithm  constructs  a  decision  tree  by  recursively  splitting  the  instance  space  (training 
set — e.g.,  the  SAS-076  ship  data  set).  Let  T  be  the  set  of  training  data  that  reaches  a  particular  node.  A  splitting 
criterion  is  used  to  determine  which  attribute,  and  what  value  of  that  attribute,  is  optimal  to  split  T  into  sets  Ti 
and  TR.  The  splitting  criterion  is  based  on  minimizing  the  standard  deviation  of  the  known  class  values  (e.g., 
ship  cost)  in  T .  The  expected  error  reduction,  or  standard  deviation  reduction  (SDR),  is  calculated  by 

SDR  =  sd(T)  -  x  sd(TL)  -  ^  x  sd(TR)^j  ,  (15) 

where  sd(T)  is  the  standard  deviation  of  the  class  values  in  T . 

The  recursion  stops  at  a  particular  node  when  the  standard  deviation  is  only  a  small  fraction  (e.g.,  5%)  of 
the  standard  deviation  of  the  original  instance  set  (at  the  root  of  the  tree).  Splitting  also  terminates  when  only  a 
few  instances  remain  (T  <  4)16. 


Fitting  Linear  Regression  Models 

After  the  tree  has  been  grown,  the  M5  system  computes  a  linear  regression  model  for  every  node  N  of  the  tree, 
recursively  descending  the  tree.  Each  regression  model  has  the  form 

wo  +  wifli  +  W2a2  +  ■■■  +  Wkak,  (16) 


where  a\,a2,  ■  ■  ■  ,a^  are  (numeric)  attribute  values.  The  weights  tv i ,  tv? •  ■  •  ■  ■  wy  are  calculated  using  standard 
multivariate  linear  regression.  Only  the  attributes  that  are  used  for  splitting  decisions  in  the  subtree  below 
a  node  are  used  in  the  regression.  The  other  attributes  that  affect  the  prediction  value  have  been  taken  into 
account  in  the  tests  that  lead  to  that  node.  Attributes  are  greedily  dropped  from  a  regression  model  if  doing  so 
improves  the  expected  estimated  error,  computed  as 


/  \  n  +  v  ^instances  deviation  from  predicted  class  values 

Error(N)  = - X - 

n  —  v  n 


(17) 


where  n  is  the  number  of  instances  at  the  node  and  v  is  the  number  of  parameters  in  the  node’s  regression  model. 
This  expected  estimation  error  is  calculated  by  averaging  the  absolute  difference  between  the  predicted  value 
and  the  actual  value  for  each  of  the  training  examples  that  reach  that  node.  This  results  in  underestimation 
of  the  expected  error  outside  the  calibrating  data.  As  a  compensation  factor,  the  expected  error  is  multiplied 
by  (n  +  v)/(n  —  v).  Dropping  a  term  decreases  the  latter  multiplication  factor,  which  may  be  enough  to  offset 
the  increase  in  the  average  error  over  the  training  cases.  This  procedure  is  similar  to  the  modification  of  the 
coefficient  of  determination,  R2,  to  the  more  representative  R adjusted  'n  regression  theory  [13]. 

1 ’’Experiments  have  shown  that  the  results  obtained  are  not  very  sensitive  to  the  exact  choice  of  these  thresholds. 
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Pruning  the  Tree 

Once  the  tree  is  built  and  linear  regression  models  fitted  for  each  node,  the  tree  is  recursively  pruned  from  the 
leaves  if  this  results  in  a  lower  expected  estimated  error.  If  T  is  the  set  of  instances  that  reach  a  node,  let  the  Tr 
and  Tr  be  the  split  of  these  instances  between  the  left  (L)  and  right  (R)  children  nodes.  For  each  node  N,  the 
error  calculated  by  equation  (17)  is  compared  to  the  expected  error  from  the  subtree  below  (branches  L  and  R), 
recursively  calculated  as 

I  Til  X  subtreeError(L)  +  |7/d  x  subtreeErroriR) 
subtreeError(N)  = - — - if  N  is  an  internal  node, 

subtreeError(N)  =  Error(N)  if  N  is  a  leaf.  (18) 


Smoothing  the  M5  Model  Tree 


After  pruning,  the  adjacent  linear  models  will  be  sharply  discontinuous  at  the  leaves  of  the  pruned  tree.  This 
problem  is  pronounced  for  models  constructed  from  a  small  number  of  training  instances.  The  M5  system 
applies  a  smoothing  process  combining  the  linear  regression  model  at  a  leaf  with  the  models  on  the  path  to  the 
root  to  form  the  final  model  that  is  placed  at  the  leaf.  In  effect,  the  estimated  value  of  the  leaf  model  is  filtered 
along  the  path  back  to  the  root.  At  each  node,  that  value  is  combined  with  the  value  predicted  by  the  linear 
model  for  that  node.  The  calculation  is 


/ 

P  = 


np+kq 
n  +  k  ’ 


(19) 


where  p'  is  the  prediction  passed  up  to  the  next  higher  node,  p  is  the  prediction  passed  to  this  node  from  below,  q 
is  the  value  predicted  by  the  model  at  this  node,  n  is  the  number  of  training  instances  that  reach  the  node  below, 
and  k  is  a  constant.  Experiments  by  Wang  and  Witten  [15]  showed  that  smoothing  substantially  increases  the 
accuracy  of  predictions.  After  the  smoothing  process,  the  M5  model  tree  effectively  has  constructed  piece-wise 
linear  regression  models. 


Nominal  Attributes 

Nominal  attributes  are  transformed  into  binary  variables  and  then  treated  as  numeric  during  the  construction  of 
the  model  tree.  For  each  nominal  attribute,  the  average  class  value  (e.g.,  cost)  corresponding  to  each  possible 
attribute  value  is  calculated  from  the  training  set.  The  attribute  values  are  then  sorted  from  smallest  to  largest 
average  class  value.  A  nominal  attribute  with  k  possible  values  is  converted  into  k  —  I  synthetic  binary  variables: 
the  ith  being  0  if  the  original  nominal  value  is  one  of  the  first  i  in  the  sorted  ordering  and  1  otherwise. 


Missing  Values 


To  handle  training  instances  with  missing  attribute  values,  the  standard  deviation  reduction  formula  is  modified 
to  compensate  for  missing  values.  For  an  attribute  for  which  m  >  1  instances  are  missing,  the  formula,  (15), 
re-written  as 


SDR  = 


m 

T 


x 


sd(T) 


x  sd(TL) 


!%[ 

in 


x  scI(Tr) 


(20) 


For  instances  with  missing  attribute  values,  a  surrogate  value  is  used  during  the  construction  of  the  model 
tree.  The  missing  value  is  replaced  with  the  average  value  of  that  attribute  over  the  training  instances  that 
reach  a  particular  node.  Similarly,  at  the  leaf  level  all  missing  values  are  replaced  by  the  average  values  of  the 
corresponding  attributes  of  the  training  instances  reaching  the  leaf. 
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1.0  INTRODUCTION 

1.1  Background 

SAS-076  set  out  to  generate  independent  estimates  of  the  Operation  &  Support  (O&S)  costs  of  the  Netherland’s 
Her  Majesty  Ship  (HMS)  Rotterdam  Landing  Platform  Dock  (LPD).  This  was  an  ex-ante  analysis  based  on  real 
organizational  and  operational  data,  followed-up  with  technical  and  economic  data  gathered  from  the  Royal 
Netherlands  Navy  (RNLN).  SAS-076  also  generated  an  independent  cost  estimate  (ICE)  of  the  development 
and  construction  costs,  labelled  herein  as  acquisitions  costs,  for  the  LPD,  which  is  described  in  Annex  B1  of 
the  SAS-076  report.  The  reader  is  referred  to  Annex  B1  for  technical  data  related  to  the  HMS  Rotterdam  -  the 
System  of  Interest  (Sol)  for  this  study. 

For  the  selected  systems,  assumptions  were  made,  technical  data  was  gathered,  and  actual  cost  data  was 
collected  for  the  development,  production  and  in-service  phases.  SAS-076  generated  an  independent  cost  es¬ 
timate  based  on  the  SAS-054  guidelines  [1]  to  estimate  the  O&S  costs  using  3  methods:  parametric,  analogy 
and  engineering  “bottom-up”  methods.  As  expected,  the  latter  was  the  most  demanding  in  terms  of  resources 
and  work-hours  consumed. 

The  parametric  and  analogy  method  were  used  to  develop  an  O&S  cost  estimate  based  on  Cost  Analysis  Im¬ 
provement  Group  (CAIG)1  elements.  This  study  was  performed  outside  the  SAS-076  by  a  group  of  researchers 
from  the  United  States  (U.S.)  Naval  Center  for  Cost  Analysis  (NCCA).  The  results  of  this  study  are  attached  to 
this  report  in  Appendix  2.  The  engineering  method  estimates  the  O&S  costs  only  for  HMS  Rotterdam,  because 
logistic  data  was  only  available  for  this  ship. 
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1.2  Objective 

The  objective  of  this  report  is  to  present  the  methods  used,  compare  the  results  obtained,  and  describe  the  appli¬ 
cation  of  the  engineering  method  for  this  study,  as  well  as  the  benefits  obtained  and  the  difficulties  encountered. 


1.3  Scope 

This  report  focuses  mainly  on  the  engineering  method  and  how  this  should  be  used  according  to  SAS-054 
recommendations.  As  known,  this  method  is  not  new  in  estimating  costs,  but  remains  a  fundamental  one. 


1.4  Outline 

Section  2  presents  the  methods  used  in  this  study;  Section  3  presents  the  tools  used  to  compute  the  O&S 
estimates;  Section  4  defines  the  data  and  the  assumptions;  Section  5  describes  the  cost  normalization;  Section 
6  presents  the  results  of  the  analysis;  Section  7  makes  a  comparison  between  the  CAIG  and  the  Generic  Cost 
Breakdown  Structure  (GCBS)  models;  Section  8  provides  the  lessons  learned;,  and,  Section  9  presents  the 
conclusions.  In  the  Appendices  the  CATLOC2  model  is  described,  the  CAIG  study  is  presented  as  well  as  the 
Cost  Breakdown  Structures  used  in  both  the  CAIG  and  GCBS  model. 


*CAIG  is  now  replaced  by  Cost  Assessment  and  Program  Evaluation  (CAPE). 

2Life  cycle  cost  software  from  Systecon  (http://www.systecon.co.uk/products/catloc/index.asp). 
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2.0  METHODOLOGY 

The  NATO  RTO-TR-SAS-054  report  recommended  the  use  of  more  than  one  method.  In  this  study,  two  inde¬ 
pendent  O&S  cost  estimates  were  produced  using  3  methods:  parametric,  analogy  and  engineering  “bottom- 
up”  methods. 

2.1  Analogy  Method 

The  analogy  method  compares  a  new  system  with  one  or  more  existing  systems  for  which  accurate  cost  and 
technical  data  are  available.  The  historic  system  should  be  of  similar  size,  complexity  and  scope. 

2.2  Parametric  Method 

Parametric  methods  are  based  on  various  characteristics  or  measurable  attributes  of  the  system,  hardware  and 
software  being  estimated.  It  depends  upon  the  existence  of  a  causal  relationship  between  system  costs  and  these 
parameters.  Such  relationships,  known  as  Cost  Estimating  Relationships  (CERs),  are  typically  estimated  from 
historical  data  using  statistical  techniques.  If  such  a  relationship  can  be  established,  the  CER  will  capture  the 
relationship  in  mathematical  terms,  relating  cost  as  the  dependent  variable  to  one  or  more  independent  variables. 

2.3  Engineering  “Bottom-up”  Method 

Engineering  methods,  as  stipulated  in  the  SAS-054  report,  are  the  most  detailed  of  all  the  techniques  and  the 
most  costly  to  implement.  This  technique  starts  at  the  lowest  level  of  definable  work  within  the  cost  breakdown 
structure  and  builds  up  to  a  total  cost.  These  types  of  estimates  are  used  when  detailed  design  data  (Logistic 
Support  Analysis  Records)  are  available  on  the  Sol. 

Parametric  and  analogy  methods  were  applied  by  experts  from  the  U.S.  Naval  Centre  for  Cost  Analysis  and 
were  based  on  U.S.  ship  data  pulled  from  the  U.S.  Naval  Visibility  and  Management  of  Operating  and  Support 
Costs  (VAMOSC)  management  information  system  and  averaged  appropriately.  These  methods  were  applied 
on  the  U.S.  Department  of  Defense  (DoD)  CAIG  Cost  Breakdown  Structure  (CBS).  The  cost  obtained  is  the 
average  annual  O&S  costs  of  a  hull,  not  the  specific  O&S  costs  in  a  given  year  (see  details  in  Appendix  2). 

The  Engineering  bottom-up  method  was  applied  using  data  from  the  RNLN  and  was  based  on  the  CBS 
recommended  in  SAS-054,  labelled  herein  as  the  GCBS.  The  results  of  applying  this  method  can  be  found  in 
Section  7.  SAS-076  focused  on  the  engineering  method  because  it  provided  some  key  advantages: 

•  This  type  of  method  is  a  measure  related  to  given  requirements,  and  different  requirements  lead  to  differ¬ 
ent  cost  figures.  It  is  also  highlighting  the  critical  aspects  in  the  Sol  design  and  its  logistical  organization, 
which  also  makes  it  a  tool  for  project  management  and  systems  engineering. 

•  It  provides  a  structured  way  of  weighing  significant  technical  and  cost  inputs. 

•  Even  if  the  application  of  the  method  is  expensive,  its  benefits  may  outweigh  its  cost3. 

•  The  application  of  the  method  shows  the  economical  consequences  of  the  technical  system  properties 
over  time,  which  provides  the  means  of  evaluating  the  cost  implication  of  a  proposed  system  solution 

3We  should  remember  the  saying  credited  to  John  Ruston:  “It’s  unwise  to  pay  too  much,  but  it’s  foolish  to  spend  too  little”  -  this  is 
the  operating  principle  of  the  engineering  method. 
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3.0  TOOLS 

The  tool  used  for  the  complete  GCBS  cost  model  was  CATLOC.  CATLOC  is  a  tool  that  supports  all  the 
primary  uses  of  Life  Cycle  Costing  (LCC)  presented  in  the  NATO  RTO-TR-SAS-054  report.  Different  uses  of 
LCC  typically  require  different  LCC  models,  which  can  be  implemented  in  CATLOC. 

The  most  fundamental  task  in  building  an  LCC  model  is  to  define  the  costs  to  be  included  in  the  model,  and 
how  to  calculate  them.  LCC  is  broken-down  in  a  cost  breakdown  structure,  an  LCC-tree,  where  it  is  expressed 
as  the  sum  of  sub  costs  or  cost  aggregates.  These  sub  costs  are  recursively  broken  down,  until  at  the  lowest  level 
the  cost  atoms  are  found.  Cost  atoms  are  costs  that  cannot  be  expressed  as  the  sum  of  other  costs;  instead  they 
are  defined  more  explicitly  in  terms  of  parameters,  constants  and  mathematical  functions.  Thus  the  only  costs 
in  CATLOC  calculated  through  a  formula  arc  the  cost  atoms.  The  aggregated  costs  and  the  cost  breakdown 
structure  are  used  only  to  organize  and  group  the  cost  atoms  into  logical  groups.  The  purpose  of  grouping 
cost  atoms  into  cost  aggregates  is  to  improve  the  possibilities  for  analyzing  and  understanding  the  calculated 
costs.  For  a  given  set  of  cost  atoms  there  will  exist  many  logical  ways  to  group  the  cost  atoms  together.  It  will 
therefore  be  possible  to  create  different  cost  breakdown  structures,  or  cost  views. 

Based  on  the  principles  defined  in  NATO  RTO-TR-058  (SAS-028)  [2]  and  NATO  RTO-TR-SAS-054  re¬ 
ports,  a  cost  element  (cost  atom)  shall  be  defined  as  follows:  a  cost  is  due  to  the  use  of  a  Resource  to  perform 
an  Activity  for  a  Product  over  Time  for  a  defined  Customer.  This  means  that  a  cost  atom  shall  be  defined  in  five 
dimensions:  Resource  -  Activity  -  Product  -  Time  -  Customer. 

CATLOC  allows  the  implementation  of  this  principle.  As  a  matter  of  fact,  the  tool  offers  functionalities  that 
define  cost  atoms  and  parameters  in  five  dimensions: 

•  MATERIAL  (Product)  e.g.,  systems  and  items  in  the  material  breakdown  structure; 

•  TASK  (Activity)  e.g.,  corrective  and  preventive  tasks  performed  on  the  technical  system; 

•  RESOURCE  e.g.,  resources  required  for  operation  and  maintenance  of  the  systems; 

•  STATION  (Customer)  e.g.,  stations  where  systems  are  operated  and  maintenance  is  performed;  and, 

•  TIME  e.g.,  the  time  periods  of  interest  -  “life’s  length”. 

The  costs  can  be  defined  for  a  combination  of  the  dimensions.  A  subset  of  the  members  in  any  dimension  is 
called  a  domain.  The  five  dimensions  span  a  five  dimensional  space,  and  cost  atoms  and  parameters  are  defined 
upon  subsets  of  this  five  dimensional  space.  These  subsets  are  called  domains  of  definitions.  The  domain  of 
definition  for  a  cost  atom  or  parameter  is  specified  by  giving  a  domain  for  the  five  dimensions.  In  each  of  the 
five  dimensions,  except  the  time  dimension,  it  is  possible  to  build  a  dimensional  structure,  i.e.,  define  child 
objects  and  mother  objects.  The  structure  in  the  time  dimension  is  implicitly  defined  by  the  order  data  is  given 
in  the  time  table. 

The  brief  description  above  may  not  be  easy  to  understand.  For  more  details  about  the  tool  please  refer  to 
Systecon  AB,  Sweden,  the  developer  and  owner  of  this  tool. 

For  the  tools  used  in  ex  post  testing  parametric  and  analogy  method  see  Appendix  2. 

4.0  DATA  AND  ASSUMPTIONS 

This  chapter  describes  the  data  assumptions  that  were  made  in  order  to  compute  a  cost  estimate  through  an 
engineering  approach.  Most  of  the  assumptions  are  based  on  information  received  from  the  RNLN,  but  in  those 
instances  where  the  data  was  not  available  an  assumption  was  made  to  obtain  a  complete  estimate. 
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Some  of  the  parameters  used  in  this  Annex  are  described  here.  These  are  mainly  the  parameters  connected 
to  the  general  assumptions  such  as  operational  profile  for  the  Sol  and  its  maintenance  organization.  Parameters 
related  to  the  reliability,  maintainability  and  supportability  of  the  system  are  also  described  here.  These  para¬ 
meters  were  assumed  to  be  found  in  an  existing  database.  The  rest  of  the  parameters  are  presented  in  Appendix 
1. 

The  abbreviations  or  designations  used  for  all  the  parameters  are  at  the  discretion  of  the  model  designer. 
No  standards  exist.  Because  of  the  large  number  of  abbreviations  and  their  subjective  character,  an  overview 
with  all  the  acronyms  used  is  not  provided.  The  only  references  to  the  acronyms  are  found  in  this  chapter,  the 
following  chapter,  and  Appendix  1 . 

4.1  Analogy  and  Parametric 

For  the  analogy  and  parametric  methods,  the  data  captured  represents  initially  10  U.S.  ships.  Ultimately,  4 
of  them  were  dropped  from  the  dataset  because  the  dimensions  of  those  classes  were  not  analogous  to  the 
Rotterdam  and  Johan  de  Witt  classes.  One  was  dropped  because  there  were  only  two  years  of  data  available  for 
that  class.  Therefore,  5  ship  classes  were  used  in  the  final  dataset. 

Sources  are  the  U.S.  Naval  VAMOSC  management  information  system.  VAMOSC  collects  and  reports  U.S. 
Navy  and  U.S.  Marine  Corps  historical  weapon  system  O&S  costs.  VAMOSC  provides  the  direct  O&S  costs  of 
weapon  systems,  some  linked  indirect  costs  (e.g.,  ship  depot  overhead),  and  related  non-cost  information  such 
as  flying  hour  metrics,  steaming  hours,  age  of  aircraft,  etc. 

4.2  Engineering  Bottom-up 

For  this  assessment  the  O&S  costs  are  estimated  using  both  organizational  and  operational  data  as  well  as 
technical  and  economical  data.  Data  was  gathered  by  the  RNLN  and  consists  of  the  information  described 
below.  The  level  of  confidence  of  the  O&S  cost  is  considered  high  when  data  was  provided  from  national 
sources,  and  medium  or  low  when  provided  from  public  available  data. 

However  even  data  obtained  from  RNLN  can  be  considered  uncertain  most  of  the  time  because  the  data 
that  the  group  asked  for  was  not  provided  and  thus  more  calculations  and/or  approximations  were  done  which 
inevitably  induced  more  errors.  Furthermore,  the  RNLN  could  not  provide  the  group  with  all  necessary  data, 
mostly  because  of  confidentiality  reasons,  which  led  to  the  use  of  publicly  available  data.  In  general,  the 
results  of  the  computation  are  considered  to  be  of  medium  level  of  confidence.  However,  some  cost  atoms  are 
considered  to  be  of  low  level  of  confidence. 

Each  section  below  has  a  short  description  of  the  data  and  assumptions  in  qualitative  and,  whenever  possible, 
in  quantitative  terms.  Each  section  also  includes  a  description  of  the  parameters  used  in  the  cost  model,  and 
refers  to  a  type  of  data  or  assumption.  Some  of  the  parameters  and/or  assumptions  are  only  briefly  described 
here.  A  more  detailed  explanation  can  be  found  in  Appendix  1 . 

4.2.1  Organizational  and  Operational  Data 

Organizational  and  operational  data  and  assumptions  describe  the  operational  profile,  support  organization  of 
all  stakeholders  including  the  interfaces,  and  the  administrative  and  logistic  delay  times  according  to  national 
doctrines.  It  records  also  the  requirements  of  the  Nations  in  terms  of  measures  of  effectiveness.  This  section 
also  describes  data  and  assumptions  related  to  the  programme  management  organization,  production  organiza¬ 
tion,  and  test  and  trials  organization. 
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4.2. 1.1  Operational  usage  and  types  of  missions 

Operational  usage  defines  the  average  operational  time  per  system  and  year  in  hours.  In  the  study  it  is 
assumed  that  the  following  operational  profile  is  valid  for  the  system’s  life  cycle: 

•  War/crisis  periods  (when  the  system  is  assumed  to  be  at  sea)  are  defined  in  the  model  as  TOW  (average 
war/crisis  operational  time  per  system  in  hours  per  year). 

•  SAR  missions  (when  the  system  is  supposed  to  be  at  sea)  are  defined  in  the  model  as  TOS  (average  SAR 
operational  time  per  system  in  hours  per  year). 

•  Periods  of  time  spent  in  foreign  harbour,  defined  in  the  model  as  TOH  (average  operating  time  spent  in 
harbour  elsewhere  in  hours  per  year). 

•  Periods  of  time  spent  at  home  harbour,  defined  in  the  model  as  TSB  (average  operating  time  stand-by  in 
hours  per  year). 

These  4  types  of  operational  time  periods  define  the  mission  types  and  can  be  developed  in  more  details  if 
necessary.  The  first  three  periods  above  are  considered  as  operational  time  and  can  be  expressed  as: 

TO  =  TOW  +  TOS  +  TOH,  (1) 

where  TO  is  the  average  operational  time  per  system  per  year,  and  is  calculated  as  the  sum  of  the  other  opera¬ 
tional  periods  of  time. 

The  model  defines  a  period  of  time  when  the  system  is  supposed  to  be  at  the  contractor  site  for  maintenance 
actions,  TSS  -  average  time  stand-still  in  hours  per  year.  This  means  that  the  sum  of  these  periods  of  time  and 
TSB  (average  operating  time  stand-by  in  hours  per  year)  should  be  equal  to  the  total  number  of  hours  per  year, 
i.e., 
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8760  =  TO  +  TSB  +  TSS  .  (2) 

All  these  periods  of  time  have  been  deduced  and  computed  from  10  years  of  Rotterdam  class  ship  oper¬ 
ational  usage.  In  the  model,  which  considers  a  30  years  life  cycle  for  the  system,  an  average  value  for  the 
operational  periods  was  computed  for  the  missing  data. 

The  input  data  from  the  RNLN  was  given  for  the  time  period  between  2000  and  2009.  The  input  parameters 
used  are: 

•  NSD  -  the  number  of  sailings  days  per  year; 

•  NDHH  -  the  number  of  days  at  home  harbour,  not  including  the  time  for  major  maintenance  actions; 

•  NDFH  -  the  number  of  days  in  foreign  harbours  or  elsewhere  (not  at  home  harbour);  and, 

•  NHV  -  the  number  of  harbour  visits  elsewhere.  The  parameter  NHV  is  not  used  in  the  model; 

Using  the  input  parameters,  the  periods  of  time  defined  above  were  calculated  as  follows: 


TOW  =  NSD  x  24/2  , 

(3) 

TOS  =  NSD  x  24/2, 

(4) 
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TOH  =  NDFH  x  24  , 

(5) 

TSB  =  NDHH  x  24  , 

(6) 

TSS  =  8760  -  TO  -  TSB  . 

(V) 

The  model  defines  with  these  periods  of  time  two  other  parameters: 

•  UTILF  -  is  a  utilization  factor  describing  the  average  fraction  of  operational  time  (TO)  per  calendar  time. 

UTILF  =  TO/8760.  (8) 

•  UTILS  -  is  the  utilization  factor  describing  the  average  fraction  operational  time  at  sea  per  calendar  time, 
calculated  only  over  TOW  and  TOS 


UTILS  =  (TOW  +  TOS)/8760.  (9) 

The  input  values  obtained  from  RNLN  led  to  some  uncertainties  which  still  have  not  been  resolved,  i.e.,  it 
is  unclear  if  NDFH  includes  only  visits  to  foreign  harbours  or  visits  to  the  main  contractor’s  docks.  It  is  also 
unspecified  if  any  major  maintenance  periods  are  included  here.  In  the  model,  the  assumption  is  that  it  does. 
This  does  not  in  any  way  affect  the  costs  computation,  at  least  not  at  the  level  of  detail  used  in  the  model. 

The  operational  profile  is  still  very  simple.  In  order  to  get  all  the  benefits  from  the  engineering  method  the 
definition  of  the  operational  profile  and  different  types  of  missions  should  be  described  in  much  more  detail. 

4.2. 1.2  Types  of  operators 

According  to  the  information  received  from  RNLN,  the  manning  profile  is  given  in  Table  1 . 

Assumptions  used  in  Table  1  are: 


Table  1 :  Rotterdam  class  ship  manning  profile 


Number  of  Accommodations 

Number  of  Crew  Members 

Commander 

1 

1 

Chief  Officer 

1 

1 

Officers 

14 

11 

CPO’s 

28 

22 

Enlisted 

84 

78 

Total 

128 

113 

•  The  total  number  of  128  includes  the  reserves  and  equals  the  total  number  of  accommodations  on  board 
the  ship.  The  total  of  1 13  is  the  actual  number  of  crew  members.  It  is  considered  that  the  number 
of  persons  included  in  the  crew  list,  i.e.,  113  will  not  change  with  the  operational  profile.  The  model 
considers  the  total  number  of  persons  including  the  reserves,  i.e.,  128. 
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•  The  costs  related  to  the  operating  personnel  on-board  the  ship  is  an  average  income  per  year,  MIOP,  i.e., 
50.000  Euros, 

•  The  model  gives  the  possibility  to  implement  more  detailed  costs  related  to  the  average  yearly  cost  for 
personnel  uniforms,  accessories,  shoes,  special  clothing,  etc.  through  the  parameter  COPC,  but  the  data 
was  not  available  and  this  parameter  is  set  to  zero  for  the  estimation. 

•  For  the  other  types  of  personnel,  the  costs  are  captured  at  Maintenance  Costs  in  association  with  either 
corrective  maintenance  actions  or  preventive  maintenance  actions. 

•  No  increase  in  pay,  inflation  rate  or  present  value  factor  is  applied  to  the  salaries. 

•  Because  of  the  lack  of  data,  other  parameters  such  as  operator  tasks,  i.e.,  categories  of  tasks  necessary  to 
operate  the  ship,  or  the  quantity  of  personnel  necessary  to  operate  the  ship,  type  of  personnel  and  task, 
as  well  as  the  percentage  of  the  working  time  dedicated  to  the  operation  of  the  ship  per  type  of  personnel 
and  task  category,  were  not  included. 

•  According  to  RNLN  there  is  a  personnel  cost  associated  to  the  helicopters  operating  from  the  ship.  The 
data  received  from  the  RNLN  is  an  average  cost  per  year  and  is  connected  to  the  total  personnel  costs  for 
the  ship. 

4.2. 1.3  Consumables  and  all  fluids 

These  assumptions  reflect  technical  and  economical  data  on  fuel,  oil  and  lubricants  consumption,  as  well  as 
other  consumables,  except  repair  parts.  These  consumables  can  be  anything  from  office  and  personal  computer 
supplies  to  water  or  other  fluids,  food  and  housekeeping  materials. 

4.2. 1.3.1  Fuel 

The  input  data  from  the  RNLN  contained  the  fuel  consumption  in  litres  per  year  for  approximately  10  years. 
A  conversion  from  cubic  metres  to  metric  ton  was  made.  According  to  the  Shell  Oil  Company,  the  density  for 
maritime  fuel-oil  is  between  840  -  990  Kg/m3,  depending  on  the  type  of  fuel.  The  average  density  chosen  was 
915  Kg/m3,  leading  to  the  following  density  coefficient  used  for  conversion: 

915/100  =  0.92.  (10) 

This  is  an  assumption,  resting  on  the  hope  that  some  environmental  considerations  will  be  made  when 
choosing  the  fuel. 

The  BWI  (Bunkerworld  Index)  shows  the  actual  price  for  different  types  of  fuel  in  $/MT  (metric  ton).  At 
the  time  of  this  study  the  prices  varied  between  600  and  960  $/MT.  However,  the  average  price  in  the  last  five 
years  was  between  370  and  670  $/MT,  depending  on  the  type  of  fuel.  The  average  price  used  for  the  estimation 
is  370  euro/MT,  using  the  present  currency  conversion  from  $  to  euro,  -  parameter  CLFS  in  the  model. 

Bunker  Index  (BIX)  is  the  Average  Global  Bunker  Price  (AGBP)  for  all  maritime  gasoil  port  prices  pub¬ 
lished  on  the  Bunker  Index  website.  More  than  5  BIX  were  used  to  calculate  the  average  price  used  in  the 
model.  The  BIX  is  calculated  by  adding  each  individual  daily  fuel-type  port  assessment  price  and  dividing  by 
the  number  of  prices.  The  statistics  (the  last  five  years  prices)  from  the  same  source  were  used. 

The  fuel  prices  showed  huge  variations  over  the  last  few  years;  therefore  the  effect  of  this  cost  variation 
on  the  total  LCC  can  be  high,  especially  for  yearly  planning  and  budgeting.  Due  to  the  complexity  of  the 
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estimation  of  future  fuel  prices,  and  due  to  the  time  and  resources  limitations  the  group  faced,  it  has  not  been 
possible  to  calculate  the  costs  using  future  fuel  price  estimates.  However  the  user  of  the  Sol  can  use  estimations 
of  the  fluctuation  of  the  fuel  prices  from  professional  sources  and  adjust  the  training  days  at  sea  according  to 
the  budget,  or  buy  the  necessary  quantity  of  fuel  when  the  prices  are  lowest  for  future  years  according  to  the 
planned  operational  profile.  A  storage  cost  will  eventually  arise.  Of  course,  this  can  be  done  in  peace  time.  In 
war  time  or  crisis,  the  cost  of  fuel  is  still  important,  but  the  shortage  of  fuel  is  more  important  and  the  budget 
will  need  to  be  adjusted  accordingly. 

These  are  usually  the  reasons  that,  when  using  the  engineering  LCC  method,  lead  to  the  elimination  of  these 
variations  in  the  fuel  prices.  In  using  the  engineering  method,  one  goal  is  to  find  the  cost  drivers  which  then  can 
be  reduced  or  affected  by  the  user  of  the  system  himself. 

The  average  consumption  (expressed  in  the  model  by  MFCH)  for  the  given  10  years  is  3461  m3  or  3,461,000 
litres  per  year.  Assuming  that  the  cost  for  fuel  is  in  average  370  euro/MT,  then  the  average  cost  for  fuel  con¬ 
sumption  is  much  higher  than  the  average  value  given  by  the  RNLN  for  yearly  petroleum,  oil,  and  lubricant 
(POL)  consumption.  The  model  assumes  that  the  cost  for  the  fuel  consumption  is  not  included  in  this  value  and 
an  additional  cost  is  calculated. 

4.2. 1.3.2  Oil  and  lubricants 

In  some  publications,  the  oil  consumption  is  said  to  be  100  times  smaller  than  the  fuel  consumption  at 
nominal  power,  i.e.,  about  1%.  Other  assumptions  are  that  for  large  ships  the  oil  consumption  is  larger  than 
10%  of  fuel  consumption.  In  lack  of  actual  follow-up  data,  it  has  not  been  possible  to  find  reliable  information 
to  make  anything  but  a  rough  estimation  for  the  oil  consumption. 

In  the  model,  a  formula  is  used  which  calculates  the  oil  consumption  as  15%  of  the  fuel  consumption. 
The  propulsion  systems  for  Rotterdam,  both  primary  and  secondary  systems,  are  diesel  generators  and  motors. 
According  to  information  gathered  from  specialized  literature,  diesel  generators  and  motors  have  a  rate  of  lu¬ 
bricants  consumption  that  may  approach  5%  of  the  fuel  consumption.  This  is  the  assumption  used  in  the  model. 

4.2. 1.3.3  Food  and  water 

Reasonable  water  consumption  level  per  person  is  about  150  litres/day.  There  are  128  people  on  the  ship 
and  they  spend  about  200  days  on-board  on  average  which  gives  a  total  water  consumption  of  3,840,000  litres 
or  3,840  m3  per  year.  In  The  Netherlands,  the  water  price  is  on  average  2.6  euro/m3.  This  results  in  about 
10,000  euro/year  only  for  personnel  needs.  We  estimate  that  the  minimum  needs  for  water  are  at  least  3  times 
higher. 

No  increase  rate  has  been  applied  to  the  prices  although  there  has  been  a  larger  increase  in  the  price  of  water 
in  the  Netherlands  than  in  many  European  countries.  Organisation  for  Economic  Co-operation  and  Develop¬ 
ment  (OECD)4  average  real  increase  (%/year)  was  4.6  between  1990  and  1998. 

4.2. 1.3.4  Other  consumables 

In  the  model,  other  consumables,  such  as  the  electricity  consumption  and  other  operational  materials  could 
be  calculated.  As  no  input  data  was  obtained,  those  elements  had  no  impact  on  the  O&S  costs. 


4www.oecd.org 


RTO-TR-SAS-076 


B2  -9 


ANNEX  B2  -  ESTIMATING  THE  OPERATION  AND 
SUPPORT  COSTS  OF  THE  ROYAL  NETHERLANDS  NAVY 
LANDING  PLATFORM  DOCK  SHIPS:  AN  EX  ANTE  ANALYSIS 


ORGANIZATION 


4.2. 1.3  Munitions  and  countermeasures 

Data  supplied  by  the  RNLN  included  an  average  cost  per  year  for  use  of  munitions  and  countermeasures 
for  training  and  missions.  No  other  assumptions  are  made. 

4.2. 1.4  Support  concept  and  organization 

For  maintenance  tasks,  the  following  general  maintenance  levels  are  distinguished: 

•  Organizational  Level  Maintenance  (OLM) 

•  Intermediate  Level  Maintenance  (ILM) 

•  Depot  Level  Maintenance  (DLM) 

•  Contractor  Level  Maintenance  (CLM) 

C  L  M 


I  L  M 


DLM 


OLM 


1LPD 

Figure  1 :  General  maintenance  levels 


OLM  -  level 

OLM-level  consists  of  repairing  failures  by  exchanging  Line  Replaceable  Units  (LRUs),  preventive  mainte¬ 
nance,  corrective  maintenance  and  condition  monitoring.  This  is  done  by  the  crew  of  the  ship,  totally  indepen¬ 
dent  of  any  support  from  ashore.  Basically  the  activities  take  place  at  sea  or  in  the  harbour.  The  maintenance 
personnel  make  use  of  tools,  support  equipment,  spare  parts  and  documentation  on  board  the  ship. 
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ILM/DLM  -  levels 

Personnel  employed  at  these  levels  are  not  specified  in  the  model  and  subsequently  no  average  income  per 
year  or  month  is  specified  either.  The  only  cost  taken  into  consideration  here  is  the  average  man-hour  cost,  i.e., 
MHC,  65  euro/hr.  However,  the  model  gives  the  opportunity  to  specify  the  number  of  workers  used  for  each 
preventive  (PM)  or  corrective  (CM)  maintenance  action  on  each  part  in  the  materiel  structure,  as  well  as  the 
number  of  labour  hours  per  each  action. 

Maintenance  at  ILM-  and  DLM-level  is  organised  or  done  by  the  Marinebedrijf  (Navy  Maintenance  Estab¬ 
lishment,  NME).  The  NME  or  third  parties  will  do  the  complex  and/or  labour  intensive  maintenance  tasks. 

The  crew  of  the  ship  can  assist  in  the  tasks  at  these  levels  (hours  spent  included  in  the  maintenance  capacity 
at  OLM-level).  Basically  the  NME  will  do  their  maintenance  tasks  during  a  scheduled  maintenance  period. 
Planned  activities  are  done  during  the  maintenance  periods  as  captured  in  the  yearly  maintenance  plan,  as  long 
as  they  are  not  part  of  a  maintenance  period  during  a  mission.  The  NME  will  repair  the  failed  or  supposed  to 
be  failed  mechanical  and  electronic  LRUs,  which  then,  after  a  check,  will  be  added  to  the  stock. 

Statutory  tasks,  like  yearly  inspection  and  testing  of  pressurized  containers  and  lifts,  are  always  done  by  the 
NME.  The  same  counts  for  calibration  activities  on  Test  and  Support  Equipment  and  tools. 

For  the  activities  at  these  levels,  a  larger  set  of  tools  or  workshops  ashore  can  be  used.  The  electronic 
workshops  will  be  equipped  with  the  necessary  tools,  test  equipment  and  test  software,  as  far  as  it  is  considered. 

The  difference  between  these  two  levels  is  that  activities  at  the  ILM-level  level  are  characterized  by  a  short 
preparation  time. 

CLM  -  level 

Contractor  Level  Maintenance  is  the  maintenance  that  is  done  by  industry.  The  contractor  has  specialized 
knowledge  about  the  system  and  specialized  equipment  to  perform  the  maintenance  actions  as  efficiently  as 
possible.  Cost  of  contractor  logistics  support  (CLS)  captures  the  costs  for  using  highly  specialised  personnel, 
and  the  costs  of  transportation  related  to  these  maintenance  actions. 

4.2. 1.5  Logistic  Support  Elements 

Data  and  assumptions  for  logistic  support  elements  describe  parts  of  the  Integrated  Logistic  Support  Ele¬ 
ments  (LSE)  that  are  necessary  for  the  costing  study.  These  elements  are:  Documentation,  Packaging,  Handling, 
Storage  &  Transport,  Facilities,  Training  and  Training  devices,  data  exchange  and  related  information  systems, 
and  computer  resource  support. 

4. 2. 1.5.1  Documentation  update 

No  data  was  available  from  the  RNLN  if  either  investments  in  or  updates  of  the  documentation  was  needed 
to  operate  or  to  maintain  the  Sol. 

The  value  of  the  investment  costs  for  the  documentation  needed  for  a  complex  system  like  the  Rotterdam 
LPD,  can  vary  depending  on  how  much  experience  the  producer  of  such  a  system  has,  or  how  many  systems 
are  already  produced.  Assuming  that  the  manufacturer  of  the  system  has  such  experience,  but  considering  that 
only  one  system  or  very  few  have  been  produced  and  that  The  Rotterdam  class  might  have  been  one  of  the  first 
systems  delivered  to  the  customers,  we  can  estimate  the  cost  for  initial  documentation  to  around  8-10%  of  the 
total  acquisition  cost. 
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The  yearly  updates  are  on  average  10  to  15%  of  the  initial  cost,  but  may  be  more  depending  on  which 
technical  modifications  are  carried  out  on  the  main  system.  Now  included  in  this  estimation  is  administration 
and  transportation  and  any  other  costs  connected  with  the  documentation. 

This  is  a  very  rough  estimation,  more  so  than  should  be  accepted  in  a  normal  engineering  estimation. 

4.2. 1.5.2  Continuous  Training 

By  training,  we  mean  the  necessary  courses  to  teach  the  personnel  how  to  operate  the  system  and  how 
to  maintain  it.  The  type  of  parameters  considered  here  are  the  training  volume  and  types,  as  well  as  the  cost 
per  training  type.  Also  considered  is  how  many  training  devices  are  used  and  the  cost  for  acquisition  and 
maintenance  of  the  devices.  No  data  was  made  available  from  the  RNLN 

The  assumption  made  here  is  that  the  yearly  cost  for  training  is  comparable  to  the  cost  for  documentation 
updates,  or  slightly  lower.  The  value  used  is  10%  lower  than  that  for  documentation  updates.  This  simplified 
assumption  excludes  the  initial  cost  for  acquiring  the  necessary  training  devices  or  training  documentation  as 
well  as  the  initial  training  events. 

4.2. 1.5.3  PHST  ( Packing ,  Handling,  Storage,  &  Transporting) 

Data  obtained  from  the  RNLN  was: 

•  Materiel  costs  for  handling  of  Government  Owned  Stores 

•  Man-Hour  Costs  for  handling  of  Government  Owned  Stores 

Initially,  implemented  in  the  cost  model  were  parameters  like:  storage  volume,  percentage  of  storage  areas 
used  for  the  ship,  storage  costs  or  cost  per  m3,  as  well  as  the  frequency  of  transport  per  type  of  transport,  and 
for  each  type  of  material  to  be  transported:  equipment  spare  parts,  personnel,  munitions  and  consumables. 
However,  no  data  was  available  and  these  parameters  were  removed  from  the  CBS. 

N.B.:  Other  costs  such  as:  costs  for  the  support  of  the  information  systems  (hardware/software);  costs  for 
embedded  software  and  their  support;  costs  for  the  use  of  the  infrastructure  at  home  or  elsewhere  (different 
types  of  fees)  exist  in  the  model.  However,  lack  of  data  or  other  possibilities  to  estimate  them  have  made  their 
impact  on  the  total  outcome  nil. 

The  assumptions  made  for  training  and  documentation  updates  and  for  other  similar  assumptions  are  based 
on  Swedish  experience,  and  rely  on  data  bases  for  similar  costs  for  sea,  air  and  land  systems. 


4.2.2  Technical  Data 

4.2.2. 1  Life  length  of  the  system 

Expected  life  length  of  the  system  is  an  input  from  the  RNLN  and  is  equal  to  30  years.  In  CATLOC  the 
built-in  parameter,  TIMEVAL,  is  taken  into  consideration  in  the  TIME  domain  but  is  not  explicitly  used  in  any 
formulae.  More  detailed  information  can  be  found  in  the  description  of  the  model. 
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4.2.2.2  System  Breakdown  Structure 

This  ICE  used  the  generic  CBS  as  defined  in  the  SAS-054  Report.  The  cost  elements  that  were  estimated 
were  defined  iteratively.  The  list  of  the  cost  elements  for  this  study  as  well  as  their  definitions,  are  described 
in  the  next  section.  Furthermore  the  methodology  defined  in  the  SAS-028  Report  with  the  generic  resources, 
product  and  task  dimensions  were  used;  and  for  the  definition  of  the  generic  System  Breakdown  Structure, 
ANEP-41  [3]  was  used.  If  necessary,  these  approaches  were  tailored  according  the  specifics  of  the  programme. 

For  the  O&S  costs  (ex-ante  estimate),  even  if  not  all  the  aggregates  could  be  populated  with  input  data,  the 
costs  were  split  into  the  generic  System  Breakdown  Structure  as  follows: 


Group  000 

General  Guidance  and  Administration 

Group  100 

Hull  Structure 

Group  200 

Propulsion  Plant 

Group  300 

Electric  Plant 

Group  400 

Command  and  Surveillance 

Group  500 

Auxiliary  System 

Group  600 

Outfit  and  Furnishings 

Group  700 

Armament 

Group  800 

Integration/Engineering 

Group  900 

Ship  Assembly  and  Support  Services 

An  example  of  the  system  breakdown  structure  and  how  this  was  used  in  the  estimation  is  presented  in 
Figure  2  as  an  Excel  sheet,  MtrlData,  that  includes  the  reliability  data  for  each  Item  of  the  System  Breakdown 
Structure.  The  structure  can  be  adjusted  by  the  system  user. 


Information  from  KM.N 

System  Material  Data 

|Basic  Information 

MID 

DESCR 

SMID 

TYPE  QTYPM 

PRICE 

FRT 

LRE  |  MTTRP  |  NMTRP  | 

CMRP 

100 

HULL  STRUCTURE, 

MAI  N  SYS 

ASSY 

1 

o 

A  0  1 

0 

“TO 

SHELL  AUD  SUPPORTING  STRUCTURE 

100 

SUBSYS  1 

' 

o 

A  5  1 

. 

HULL  STRUCTURAL  BULKHEADS 

100 

5UBSY5  1 

1 

o 

A  0  1 

150 

HULL  UtCKS 

TO- 

SUBSYS  1 

1 

o 

A  0  1 

. 

A 

HULL  HLA LEDUMS  AN [HLAIS - 

- TO 

"StJBSVS — 1 — 

1 

0 

A  0  1 

L 

Figure  2:  Sample  System  Breakdown  Structure 

The  values  of  the  failure  rates  for  the  subsystems  should  be  either  the  sum  of  the  same  parameter  for  the 
incorporated  components  of  the  subsystems  or  a  “residual  failure  rate”.  To  explain  the  term  residual  failure  rate 
(RFRT),  the  following  example  is  given:  we  assume  that  not  all  the  components  of  a  subsystem  are  listed  in  the 
input  sheet,  therefore  the  total  failure  rate  for  the  subsystem  can  be  greater  than  the  sum  of  the  failure  rates  for 


RTO-TR-SAS-076 


B2  - 13 


ANNEX  B2  -  ESTIMATING  THE  OPERATION  AND 
SUPPORT  COSTS  OF  THE  ROYAL  NETHERLANDS  NAVY 
LANDING  PLATFORM  DOCK  SHIPS:  AN  EX  ANTE  ANALYSIS 


ORGANIZATION 


its  listed  components.  So  that  the  Total  FRT  (TFRTsubsys)  for  a  subsystem  equals  the  sum  of  the  failure  rates 
for  its  listed  components  plus  the  residual  failure  rate: 

TFRTsubsys  =  FRTlisteditem  +  RFRT.  (11) 

If  the  values  for  the  failure  rates  for  a  subsystem  are  zero,  than  the  FRT  listed  will  be  used. 

Figure  3  is  a  part  of  the  Materiel  Data  sheet  delivered  to  the  RNLN  to  till  with  follow-up  data.  The  parame¬ 
ters  seen  in  the  columns  are  described  below. 


|CM  Repairs  data 

CMRPC 

ACMRP 

CMTRP 
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Figure  3:  Input  failure  rates 


4.2.2.3  Maintenance  and  support 

4. 2. 2. 3. 1  Basic  Input 


The  parameters  in  this  section  are  the  basic  parameters  connected  to  the  reliability,  maintainability  and 
supportability  (RAMS)  data.  Most  of  these  parameters  are  well  known  from  RAMS  analysis 


•  IID  Item  Identifier  -  the  item  ID  tag  is  used  as  a  unique  identifier  per  item.  An  ID  tag  is 

defined  in  the  input  table.  The  value  should  be  entered  as  a  character  string.  There  is  no 
maximal  length  but  a  maximum  around  10-12  characters  is  recommended.  Delimiter 
characters  can  cause  trouble  when  exporting  data  to  text  format. 

•  DESCR  is  a  free  text  column  used  to  describe  an  item,  system,  materiel  position,  station,  failure 

mode,  preventive  maintenance  task  or  resource  more  in  detail. 

•  MID  Mother  item  or  System  Identifier.  It  specifies  the  ID-tag  of  the  mother  unit  in  the 

material  hierarchy.  The  mother  unit  is  either  a  system  or  another  item. 


•  TYPE  The  type  determines  whether  an  item  is  repairable,  partially  repairable  or  discardable 
and  whether  it  is  a  primary  or  secondary  item.  It  can  also  specify  that  it  is  an  assembly 
or  subsystem.  The  default  value  is  LRU  or  SRU  depending  on  whether  the  item  is 
primary  or  not. 


-  LRU  Repairable  primary  item  replaced  directly  in  the  system.  An  LRU  can  be  a  part  of 

a  subsystem,  assembly  or  can  be  a  part  of  the  main  system. 

-  SRU  Repairable  secondary  item  replaced  in  an  LRU,  SRU,  PRU  or  SPRU  at  LEVEL2 

and  3. 

-  DU  Discardable  primary  item  replaced  directly  in  the  system.  A  DU  can  be  a  part  of 

subsystem,  assembly  or  can  be  a  part  of  the  main  system. 
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-  DP  Discardable  secondary  item  replaced  in  an  LRU,  SRU,  PRU  or  SPRU  at  LEVEL2 

and  3. 

-  ASSY  An  assembly  is  ignored  in  the  logistics  breakdown,  but  it  is  used  here  to  gather  the 

costs  from  its  subsystems,  on  the  level  2  of  the  CBS.  An  ASSY  can  only  be  part  of 

the  MAINSYSTEM,  according  with  this  model  definition. 

-  SUBSYST  A  subsystem  is  ignored  in  the  logistics  breakdown,  but  it  is  used  here  to  gather  the 

cost  from  its  components,  on  the  level  3  of  the  CBS. 

-  PRU  Partially  repairable  primary  item  replaced  directly  in  the  system.  PRUs  are  still  not 

used  in  the  model.  All  PRU  must  be  converted  to  LRUs. 

-  SPRU  Partially  repairable  secondary  item  replaceable  in  a  LRU,  SRU,  PRU  or  SPRU  at 

LEVEL2  or  3  SPRU  are  still  not  used  in  the  model.  All  SPRU  must  be  converted 

into  SRUs. 

•  QTYPM  Quantity  per  Mother  item;  QTYPM  contains  the  number  of  identical  items  per  mother 

item  in  the  breakdown  structure.  The  mother  item  can  either  be  the  main  system, 
a  subsystem,  an  assembly  or  another  (repairable)  item.  The  given  value  must  be  an 
integer  >  1.  Default  value  is  1. 

•  PRICE  The  unit  price  is  an  input  parameter  given  per  item,  per  system,  per  resource  or  re¬ 

source  position.  The  value  is  mandatory  per  item  but  optional  for  systems  and  re¬ 
sources.  The  PRICE  column  is  included  in  the  input  tables  MtrlData  and  should  be 
entered  as  a  positive  float  number  in  1000  x  euro.  As  the  name  unit  price  suggests,  it 
is  assumed  that  all  prices  are  constant  irrespective  of  item,  system  or  resource  quan¬ 
tities.  The  price  used  in  the  model  do  not  includes  storage  costs.  PRICE  must  not  be 
specified  for  ASSY  items.  Any  value  specified  for  an  ASSY  is  ignored  with  a  warning. 

•  FRT  Failure  Rate  -  the  failure  rate  is  an  input  parameter  that  contains  the  number  of  failures 

per  million  hours  of  operation.  This  value  is  mandatory  per  item  and  can  also  be  given 
per  system.  It  is  given  in  the  input  table  and  should  be  entered  as  a  floating-point  value 
>  0.  The  entered  value  reflects  the  rate  per  each  individual  item/system.  The  rate  per 
item  is  not  strictly  per  failure  but  should  include  all  replacements  including  those  of 
type  “no  fault  found”,  etc.  The  total  sum  of  the  sub  item  failure  rates  must  not  exceed 
the  failure  rate  of  the  item/system  itself.  This  will  result  in  an  error  message  during  the 
calculation  test.  A  total  sub  item  failure  rate  less  than  the  FRT  of  the  item/system  itself 
is  allowed.  The  residual  FRT  in  this  case  corresponds  to  failures  that  can  be  repaired 
without  any  sub  item  replacement.  The  FRT  for  systems  is  optional.  If  not  entered 
this  value  is  calculated  as  the  sum  of  the  item  failure  rates.  FRT  must  not  be  specified 
for  items  having  TYPE  =  ASSY  or  SUBSYST.  Any  value  specified  for  an  ASSY  is 
ignored. 

4. 2. 2. 3. 2  Corrective  Maintenance  ( CM ) 

Corrective  maintenance  and  the  actions  related  to  CM  are  usually  associated  with  the  FRT/Mean  Time  Be¬ 
tween  Failure  (MTBF).  The  failure  rates  can  be  estimated  for  each  important  part  of  the  main  system.  The 
engineering  methodology  tries  to  establish  relationships  between  the  system's  failure  rates  and  the  costs  gener¬ 
ated  by  those  failure  rates.  The  model  in  CATLOC  is  trying  to  follow  the  methodology  and  gives  the  opportunity 
to  the  user  to  fill  up  the  data  available  in  different  ways.  In  the  model  there  are  two  actions  considered  to  belong 
to  the  CM:  replacement  and  repair  actions. 
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ORGANIZATION 


4.2.2.33  Replacement  data 


The  replacement  data  concerns  only  replacements  due  to  corrective  maintenance.  Generally  a  replacement 
may  be  related  to  any  replacement  action  performed  at  any  level  of  maintenance.  A  replacement  action  can  be 
a  replacement  of  a  repairable  primary  item  (i.e.,  an  LRU  or  DU)  and  in  that  case  the  replacement  takes  place  at 
OLM  or  a  replacement  of  a  faulty  repairable  secondary  item  or  discardable,  which  means  that  the  replacement 
action  takes  place  either  at  ILM  or  DLM  or  even  at  CLM. 

Usually  a  replacement  refers  only  to  replacement  actions  regarding  primary  repairable  items.  This  statement 
can  lead  to  the  impression  that  we  search  replacement  data  only  at  the  OLM  level.  This  is  only  partially  true. 
In  fact  the  model  is  not  looking  at  all  to  the  replacements  at  the  OLM  level  performed  by  the  personnel  at 
OLM  level  since  those  actions  are  included  in  crew’s  maintenance  capability  and  are  already  captured  in  the 
remuneration  of  the  crew. 

Due  to  the  nature  of  the  system,  most  certainly  some  replacements  of  important  and/or  expensive  items  can 
only  be  performed  at  ILM,  DLM,  or  CLM,  i.e.,  when  the  main  system  is  in  harbour.  Even  less  expensive  items, 
but  difficult  to  replace,  demanding  extensive  working  time  and/or  a  large  number  of  workers  can  be  interesting 
to  follow.  The  data  are  the  events  when  a  replacement  action  is  needed  to  be  performed  at  the  OLM  level,  but 
by  personnel  other  than  the  crew. 

The  input  parameters  needed  for  the  calculations  are: 


•  LRE 

•  MTTRP 

•  NMTRP 

•  CMRP 

•  CMTRP 


is  the  STID  where  the  replacement  for  a  corrective  maintenance  action  takes  place. 
Mean  time  to  replace  [h]. 

Number  of  personnel  needed  for  replacement. 

Average  cost  for  materiel  consumption  at  replacement  [thousands  of  euros]. 
Average  cost  for  transportation  due  to  replacements. 


•  ACMC  is  an  application  factor  (an  optional  column)  given  per  item  and  is  used  as  a  factor 

on  the  FRT  describing  the  average  fraction  of  the  failure  rates  which  leads  to  replace¬ 
ments  actions  performed  by  external  personnel.  This  mean  that  the  CMTC  cost  will 
be  applied  here. 

•  ACMRP  is  an  application  factor  (an  optional  column)  given  per  item  and  is  used  as  a  factor  on 

the  FRT  describing  the  average  fraction  of  the  failure  rates  which  leads  to  replacements 
actions  performed  by  personnel  from  NME  aboard,  or  at  a  foreign  harbour.  This  mean 
that  the  transportation  cost  will  be  added  here. 

•  CMRPC  is  the  average  cost  per  item  for  external  personnel  including  the  costs  for  consumables. 


•  CMRPCC  InputParameters  table  -  the  average  cost  of  consumables  (for  replacements  action  due 

to  CM)  consumption  per  year. 

•  MHCCMRP  average  man  hour  cost  for  replacement  actions  performed  by  NME. 


•  TRCCMRP  an  average  cost  value  for  transportation  events  per  year  due  to  CM  actions. 


•  CMRPEX  is  the  average  cost  per  year  for  external  personnel  including  the  cost  for  consumables. 
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If  the  LRE  is  at  the  OLM  level  and  the  replacement  action  is  performed  by  the  crew,  then  only  CMRP  should 
be  entered.  If  the  CMRP  value  is  not  known  for  each  of  the  units  replaced  on  board,  then  an  average  cost  for 
materiel  consumption  at  replacement  per  unit  of  time  at  the  level  OLM  should  be  entered  for  the  parameter 
CMRPCC. 

As  we  assumed  before,  in  case  CM  actions  are  necessary  during  the  mission  time,  not  all  replacements  at 
OLM  can  be  performed  by  the  crew.  Such  replacements  are  supposed  to  be  performed  by  the  Marinebedrijf 
(Navy  Maintenance  Establishment  -  NME)  or  by  other  external  personnel  or  contractors  hired  for  this  puipose. 

All  of  these  actions  are  understood  to  be  performed  on  board,  but  either  when  the  ship  is  at  home  harbour 
or  in  a  harbour  elsewhere.  If  the  replacement  is  performed  by  the  NME  personnel  then  at  least  parameters 
MTTRP  and  NMTRP  above  should  be  specified.  In  lack  of  data,  an  average  man-hour  cost  for  such  actions 
should  be  entered  for  the  parameter  MHCCMRP.  If  the  replacements  are  performed  by  external  personnel  or 
contractors  (LRE  =  CLM)  and  the  CMRPC  is  unknown  per  unit,  then  an  average  cost  per  year  can  be  given  to 
the  parameter  CMRPEX. 

If  the  action  takes  place  at  a  foreign  harbour,  then  an  average  cost  for  transportation  of  the  personnel  as 
well  as  for  the  unit  replaced  (back  and  forth)  can  be  specifically  given  in  the  column  CMTRR  in  the  input  sheet 
MtrlData  or  as  a  general  value  for  the  parameter  TRCCMRP  on  the  InputParameters  sheet.  The  value  in  the 
InputParameters  is  an  average  value  for  transportation  events  per  year  valid  for  all  the  units,  where  the  column 
CMTRP  has  a  Y  (yes)  and  not  a  numerical  value. 

The  model  is  computing  the  average  cost  for  replacements  per  year  according  to  the  formula: 


=  QTYPM  x  FRT  x  UTILF  x  0.00876 
x  (( CMRP+ACMC  x  ( CMTC  -  CMRP ) 

+ACMRP  x  (MHC2  x  NMTRP  x  MTTRP  +  CMTRP))  ,  (12) 


or 


=  QTYPM  x  FRT  xTOx  0.000001 

x  ((< CMRP+ACMC  x  ( CMTC  -  CMRP) 

+ACMRP  x  (MHC2  x  NMTRP  x  MTTRP  +  CMTRP))  .  (13) 

The  formula  is  calculated  per  year  for  each  unit  in  the  material  breakdown  structure.  The  results  can  then 
be  aggregated  at  a  subsystem  level  or  for  each  assembly  and  finally  for  the  entire  main  system. 

The  formula  also  takes  into  consideration  the  operational  profile  described  above.  The  results  can  then  be 
shown  per  unit/subsystem/assembly/system  and  year.  Note  that  the  formula  above  is  similar,  but  not  the  same, 
to  the  one  used  in  the  model. 

The  results  based  on  these  parameters,  give  the  user  many  possibilities  to  aggregate  the  input  data.  Not  all 
of  them  are  described  or  implemented  in  the  model,  but  there  is  however  some  aggregates  which  were  imple¬ 
mented  more  for  descriptive  reasons  because  no  data  could  be  delivered  by  the  RNLN  in  the  MtrlData  sheet. 

4.2.2. 3.4  Repair  data 

Repair  data  refers  to  the  repair  actions  performed  at  all  levels  of  maintenance  on  LRU  or  SRU.  The  data 
does  not  include  the  repair  actions  performed  during  the  scheduled  maintenance  periods.  A  repair  action  is 
mainly  regarded  here  as  a  replacement  of  an  SRU  or  DP  in  an  LRU  or  other  SRU,  which  already  was  removed 
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ORGANIZATION 


from  the  main  system.  Because  of  this  characterization,  there  are  similarities  between  replacement  and  repair 
input  data.  The  following  parameters  are  used: 


LCM  STID  of  repair  action. 


MTTR  Mean  time  to  repair  MTTR  [h].  It  is  the  turnaround  time,  in  hours  by  default,  per 

repair  action  that  does  or  does  not  require  sub  item  replacement,  i.e.  all  the  repair 
actions  triggered  by  the  failure  rate. 

NMTR  Average  number  of  personnel  required  per  repair  action. 


CMCM 

CMCMC 

CMCMCC 

MHCCMR 


Average  cost  for  material  consumption  (consumables)  during  a  corrective  maintenance 
task  [thousands  of  Euros] . 

Average  cost  for  a  repair  action  if  performed  by  contractor  /supplier  on  CLM  [thou¬ 
sands  of  Euros]. 

InputParameters  table  -the  average  cost  of  consumables  (for  replacements  action  due 
to  CM)  consumption  per  year. 

Average  man  hour  cost  for  repair  actions  performed  by  NME. 


CMCMEX  Average  cost  per  year  for  external  personnel  including  the  cost  for  consumables. 


The  description  is  similar  to  that  presented  for  replacements:  Focus  here  is  on  the  repair  actions  performed 
by  NME  or  the  contractor.  This  means  that  LCM  should  be  ILM,  DLM  or  CLM.  If  the  repair  action  is  performed 
by  NME,  then  both  a  cost  for  labour  and  materiel  consumption  will  be  computed.  Each  repair  action  can  trigger 
a  demand  for  spare  parts  in  stock.  However  such  demands  are  not  captured  in  the  model. 

If  the  repair  is  performed  by  personnel  of  the  NME,  then  at  least  parameters  MTTR  and  NMTR  above 
should  be  specified.  In  case  there  is  a  lack  of  data,  an  average  man-hour  cost  for  such  actions  should  be  entered 
for  the  parameter  MHCCMR.  If  the  CMCM  value  for  all  of  the  units  repaired  is  unknown,  an  average  cost  for 
materiel  consumption  for  repair  actions  performed  by  NME  per  suitable  unit  of  time  should  be  entered  for  the 
parameter  CMCMCC. 

If  the  repairs  are  performed  by  the  contractor  (LCM  =  CLM),  no  other  value  than  the  default  value  should 
be  entered  for  MTTR  and  NMTR.  Also  if  the  CMCMC  per  unit  is  unknown,  then  an  average  cost  per  year  can 
be  entered  for  the  parameter  CMCMEX.  CMCMC  or  CMCMEX  should  also  include  the  transportation  costs 
(back  and  forth)  for  the  repaired  unit. 

On  the  FRT,  no  application  factor,  describing  the  average  fraction  of  the  failure  rates  which  leads  to  repairs 
actions  performed  by  the  contractor,  is  used. 


4. 2. 2. 3. 5  Preventive  maintenance  data 


Major  preventive  maintenance  (PM)  periods  on  the  main  system  are  defined  in  the  input  data  as  a  3-years 
maintenance  period  and  a  6-years  maintenance  period.  These  periods  follow  the  calendar  time  during  the 
system’s  life  span  as  is  shown  in  Figure  4  below.  Cost  data  for  these  actions  was  acquired  from  the  RNLN  and 
implemented  in  the  model. 

Other  types  of  PM  can  be  decided  by  the  contractor.  These  periods  can  refer  to  the  whole  system  or  only  to 
subsystems  or  LRUs.  Information  on  such  PM  periods  can  be  entered  in  the  input  table  MtrlData. 


B2  - 18 


RTO-TR-SAS-076 


ANNEX  B2  -  ESTIMATING  THE  OPERATION  AND 
SUPPORT  COSTS  OF  THE  ROYAL  NETHERLANDS  NAVY 
LANDING  PLATFORM  DOCK  SHIPS:  AN  EX  ANTE  ANALYSIS 


Figure  4:  Preventive  maintenance  periods 


The  PM  actions  include  replacement  and  repair  actions.  But  the  model  calculates  the  cost  of  a  PM  action 
as  a  whole;  no  differentiation  is  being  made  between  replacements  and  repair's  as  in  CM  case.  The  parameters 
used  are: 

•  LPM 

STID  for  the  PM  task  -  should  be  B,  C  or  D. 

•  NYPM 

Number  of  PM  actions  per  year. 

•  TPM 

Preventive  maintenance  duration  -  gives  the  average  number  of  labour  hours  for  the 
PM  task  performed  by  one  person. 

•  NMPM 

Number  of  personnel  necessary  to  perform  the  PM  action  within  the  required  time 
frame  (TPM). 

•  CMPM 

The  average  cost  for  materiel  consumption  during  PM  task,  costs  for  replacement  of 
SRU/DP  are  not  included  [thousands  of  Euros]. 

•  CPTC 

Average  cost  for  PM  on  LRU/SRU  if  performed  by  contractor/supplier.  This  also 
includes  potential  transportation  costs  to  and  from  the  contractor’s  site  [thousands  of 
Euros], 

Many  other  costs  may  result  from  PM  actions  such  as  administrative  costs  or  transportation  costs.  In  order 
to  keep  the  model  fairly  simple,  but  still  useful  for  its  puipose,  the  choice  was  made  not  to  take  into  considera¬ 
tion  such  costs. 

4.2.2. 3.6  Support  and  test  equipment 

No  data  available.  No  assumptions  made. 

4. 2. 2. 3. 7  Spares  Parts  Consumption 

A  yearly  cost  for  replenishment  concerning  the  spare  parts  was  supplied  by  the  RNLN  and  implemented  in 
the  model. 
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ORGANIZATION 


4. 2. 2. 3. 8  Modifications  and  restorations 
No  data  available.  No  assumptions  made. 

4. 2. 2. 3. 9  Industrial  Logistic  Support 

No  data  available.  No  assumptions  made. 


4.2.3  Indirect  Support 

For  the  indirect  support  cost,  the  following  data  was  obtained  and  implemented  in  the  model: 

•  Materiel  costs  generated  by  Navy  Command  (Headquarters,  Personnel  &  Operations). 

•  Man  hour  costs  generated  by  Navy  Command  (Headquarters,  Personnel  &  Operations). 

5.0  COST  NORMALIZATION 

For  the  parametric  and  analogy  methods,  U.S.  ship  data  was  pulled  from  VAMOSC,  with  all  costs  in  constant 
fiscal  year  (FY)  2009  U.S.  Dollars.  In  order  to  deal  with  the  high  variability  of  O&S  costs  a  two  steps  approach 
was  undertaken: 

•  Data  from  VAMOSC  was  averaged  on  an  annual  basis  between  different  hulls  in  each  class; 

•  Then,  the  yearly  data  for  each  class  of  ship  was  averaged  over  the  life  of  the  hull,  in  all  cases  beginning 
in  1984  and  ending  when  the  ship  was  either  decommissioned  or  the  end  of  the  sample  period. 

For  the  engineering  estimation,  the  cost  normalization  follows  the  same  principles  as  for  the  acquisition 
cost  without  the  complexity  related  to  different  currencies.  The  costs  gathered  from  the  RNLN  are  expressed 
in  FY  2005  euros,  and  they  have  been  converted  to  FY  2008  euros  using  the  annual  average  index  for  The 
Netherlands  obtained  from  eurostat.  No  other  inflation  rates  or  other  financial  indices  were  applied  for  the  rest 
of  the  life  span  of  the  Sol  after  2008.  The  only  variations  of  the  yearly  costs  are  triggered  by  real  input  data 
obtained  from  the  RNLN  regarding  fuel  consumption  and  PM  costs. 

6.0  RESULTS  AND  ANALYSIS 

After  normalizing  the  cost  to  FY  2008  and  respecting  the  anonymity  requested  not  to  show  explicitly  the  ship’s 
O&S  costs,  the  same  fictitious  notional  common  currency  (abbreviated  NCC)  was  used  as  in  Annex  B 1  related 
to  the  Rotterdam  acquisition  costs.  This  was  also  done  to  be  able  to  include  the  estimate  for  the  acquisition  cost 
in  the  model  and  to  show  the  full  results.  The  value  used  in  the  model  is  the  actual  Rotterdam  acquisition  cost. 

Table  2  shows  the  values  expressed  in  the  same  currency  for  the  highest  cost  aggregates  and  for  the  total 
life  cycle.  The  rest  of  the  results  are  presented  in  percentage. 

Usually  an  analysis  made  through  an  LCC  engineering  estimate  comes  with  a  lot  of  details  on  the  distri¬ 
bution  of  the  costs  in  the  materiel  dimension  through  the  system  breakdown  structure  or  on  the  maintenance 
organization,  for  each  and  every  site  taken  into  consideration.  The  method  will  try  to  find  parts  of  the  system 
or  parts  of  the  organization  that  are  generating  the  highest  costs  in  order  to  find  a  solution  to  the  problem. 
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Table  2:  Total  O&S  costs 


Cost  Aggregate 

O&S  Costs  (Million  NCC) 

7.1 

Operation 

329.8 

7.2 

Supporting  operations 

24.0 

7.3 

Maintenance 

321.0 

7.4 

Sustaining  Support 

21.6 

7.5 

PHST 

204.5 

7.6 

Restoration 

0 

7.7 

Indirect  Support 

182.0 

Total  O&S  Costs 

1082.9 

At  the  same  time,  it  shows  where  the  resources  are  consumed  and  which  tasks  are  the  most  demanding.  The 
methodology  goes  hand  in  hand  with  RAMS  methods  based  on  both  optimization  and  simulations. 

Such  an  analysis  cannot  be  performed  here,  mostly  because  of  the  lack  of  data.  When  data  is  missing, 
the  result  of  an  engineering  estimate  does  not  generate  solutions,  merely  it  raises  more  questions  for  further 
analysis. 

However,  it  is  useful  to  present  the  results  and  then  to  question  them.  The  next  task  is  then  to  either  dig 
deeper  in  the  records  and  find  more  data,  or  start  a  short  term  follow-up  to  see  if  data  can  be  improved. 

With  a  tool  like  CATLOC,  an  analyst  can  dissect  the  data  in  many  ways  and  then  present  the  results  in  a 
meaningful  manner.  In  this  study,  it  was  not  possible  to  do  so,  as  the  quality  of  the  data  was  not  sufficient.  The 
results  presented  in  the  following  charts  are  at  a  high  level  of  the  CBS,  but  they  are  sufficient  to  show  some 
problems.  Figure  5  shows  the  distribution  between  the  O&S  cost  and  the  acquisitions  cost.  Percentage  values 
are  rounded  to  the  nearest  integer.  At  this  level  the  results  are  not  extraordinary. 


Figure  5:  LCC  showing  O&S  cost  versus  acquisition  costs 

Figure  6  shows  the  O&S  costs  in  more  detail.  It  can  be  noticed  that  the  cost  of  Sustaining  Support  and  Sup¬ 
porting  Operations  are  relatively  low  to  what  could  be  expected,  while  the  Indirect  Support  costs  are  relatively 
high.  Additional  analysis  and  data  would  be  required  to  understand  why  the  O&S  costs  are  distributed  over  the 
CBS  in  these  proportions. 

Looking  to  the  Operational  costs  (Figure  7),  it  is  clear  that  the  cost  of  personnel  is  too  high  when  compared 
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ORGANIZATION 


to  the  cost  for  fuel,  oil  and  other  lubricants.  Additional  analysis  and  data  would  be  required  to  understand  these 
differences. 


Figure  6:  O&S  cost  distribution 


Figure  7:  Distribution  of  operational  costs 

It  can  be  noticed  that  the  maintenance  costs  (Figure  8)  of  the  two  helicopters  on  the  ship  is  almost  as  large 
as  the  ship’s  cost  for  Preventive  Maintenance  (PM)  and  Corrective  Maintenance  (CM)  together.  Furthermore, 
7.3.3  (cost  related  to  other  operating  systems)  contains  some  uncertainties  (e.g.,  whether  the  helicopters  fuel 
cost  is  included  or  not).  Given  that  the  data  was  taken  from  the  RNLN  financial  system,  it  was  difficult  to  get 
additional  information  to  clearly  define  the  scope  of  this  cost. 

It  is  possible  to  breakdown  the  analysis  further  and  to  keep  on  asking  questions,  however,  these  questions 
should  be  asked  directly  to  the  user  of  the  system. 

Figure  9  shows  the  distribution  of  the  PM  costs  (materiel  and  personnel)  over  time:  7. 3. 1.1  (the  lower  part 
of  the  bars)  represents  man-hour  costs  and  7.3. 1.2  (the  upper  part  of  the  bars)  represents  materiel  costs.  Figure 
10  shows  the  distribution  of  cost  over  the  types  of  resources  used  in  operation. 

Finally,  this  section  presents  the  sensitivity  of  the  estimate  to  changes  in  the  parameter  values.  All  pa¬ 
rameters  are  increased  by  100%  and  the  impacts  of  that  increase  on  the  total  O&S  cost  are  shown  in  Figure 
11.  The  parameters  are  ranked  relatively  to  their  impact  on  the  O&S  cost.  It  is  noted  that  average  yearly  in¬ 
come  of  operators  (MIOP),  number  of  operators  on-board  (NOPT)  and  number  of  helicopters  (NHALO)  are 


B2  -22 


RTO-TR-SAS-076 


ANNEX  B2  -  ESTIMATING  THE  OPERATION  AND 
SUPPORT  COSTS  OF  THE  ROYAL  NETHERLANDS  NAVY 
LANDING  PLATFORM  DOCK  SHIPS:  AN  EX  ANTE  ANALYSIS 


Figure  8:  Distribution  of  maintenance  costs 
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Figure  9:  Preventive  maintenance  costs  over  time 
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Figure  10:  Distribution  of  cost  over  resource  types 


the  three  parameters  that  will  influence  total  O&S  cost  the  most,  given  an  equally  large  percentage  change,  but 
the  differences  are  not  large. 
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Figure  11 :  Sensitivity  analysis  over  all  parameters 

The  CM  actions  are  usually  a  major  cost  driver  for  the  O&S  total  cost.  Because  of  the  commercial  sensitivity 
of  data,  the  RNLN  was  not  able  to  provide  any  data  concerning  CM.  In  general,  it  is  considered  that  for  large 
military  systems,  the  acquisition  cost  stands  for  just  under  30%  of  the  total  LCC  and  the  O&S  cost  for  the  rest. 
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Usually  maintenance,  i.e.,  PM  and  CM  actions  stands  for  about  just  under  50%  of  the  O&S  cost.  Frequently  the 
costs  for  CM  are  not  bigger  than  the  costs  for  PM.  Of  course  in  the  above  figures  both  personnel  and  materiel 
cost  are  included. 

7.0  COMPARING  CAIG  AND  GCBS  MODELS 

The  main  objective  of  this  section  is  to  compare  the  top-down  approach  represented  by  the  CAIG  model, 
developed  at  the  U.S.  Naval  Center  for  Cost  Analysis,  with  the  bottom-up  approach  represented  by  the  GCBS 
model.  The  two  cost  models  resulting  from  applying  these  approaches  are  presented  in  the  appendices  1  and  2 
of  this  report.  However,  in  this  section,  a  closer  look  is  given  to  their  set  up. 

The  comparison  has  been  made  only  for  Rotterdam  since  the  GCBS  calculations  have  only  been  performed 
for  this  ship.  No  data  was  available  for  Johan  de  Witt. 

The  tasks  have  been  mainly  to  work  on  the  cost  structures  of  the  two  models  in  order  to  make  them  compa¬ 
rable.  The  main  tasks  were: 

•  Implement  the  two  cost  structures  in  CATLOC  with  some  restructuring  of  the  GCBS  model; 

•  Compare  the  results  of  the  both  approaches. 

7.1  Analysing  the  Models 

The  basic  structures  of  both  models  are  shown  in  Figure  12: 

GCBS  model  CAIG  model 


|  OSD  CAIG  086,  Systems  Operating  and  support  cost 

*  ■  1.0  UNIT-LEVEL  MANPOWER,  Unit-level  manpower  costs 
®  ■  2.0  UNIT  OPERATIONS,  Unit  operations  costs 

®  |  3.0  MAINTENANCE,  Maintenance  costs 
i|4.0  SUSTAINING  SUPPORT,  Sustaining  support  costs 

*  ■  5.0  CONTINUING  SYSTEM  IMPROVEMENTS,  Continuing  support  costs 
®  ■  6.0  INDIRECT  SUPPORT,  Indirect  support  costs 


Figure  12:  Structure  of  CAIG  and  GCBS  models 

In  order  to  achieve  the  main  objective  of  the  analysis:  to  compare  and  evaluate  the  two  cost  estimating 
methods,  the  models  needed  to  be  implemented  in  CATLOC  and  to  be  made  as  comparable  as  possible.  Given 
that  the  Cost  Breakdown  Structure  of  the  GCBS  model  was  more  granular  than  the  CAIG  model,  it  was  decided 
to  restructure  the  former  to  make  the  comparison  easier.  The  resulting  model  is  called  GCBS-CAIG  model  and 
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the  Cost  Breakdown  Structure  is  presented  in  Appendix  3.  During  the  GCBS  model  Cost  Breakdown  Structure 
restructuring,  some  assumptions  were  made.  These  assumptions  are  not  specifically  presented  here  but  can  be 
easily  understood  by  comparing  the  GCBS-CAIG  model  (the  transformed  GCBS  model)  presented  in  Appendix 
5  with  the  original  model  from  Appendix  1.  For  example: 

•  Cost  of  test  and  support  equipment  and  tools  has  been  moved  from  Maintenance  to  Sustaining  Support  to 
match  the  CAIG  model,  even  if  its  value  is  nil  in  the  original  GCBS  model. 

•  Materiel  and  Man-hour  costs  for  handling  of  Government  Owned  Stores  have  been  moved  to  Sustaining 
Support  too. 

As  for  the  implementation  of  the  CAIG  model  in  CATLOC,  the  initial  intention  was  to  implement  the  second 
Level  of  Cost  Elements  in  the  structure  hierarchy  (see  Appendix  4)  and  then  populate  these  costs.  However, 
this  was  not  achieved  due  to  the  lack  of  the  definitions  of  these  costs. 

After  the  restructuring  of  the  GCBS  model  Cost  Breakdown  Structure  and  the  implementation  of  the  CAIG 
model  in  CATLOC,  the  actual  comparison  can  be  done  by  mapping  the  cost  elements  in  the  GCBS-CAIG  model 
to  the  relevant  cost  aggregate  in  the  CAIG  model  for  which  an  estimate  exists  (see  Figure  13  for  an  example 
of  the  principle).  This  will  allow  for  an  assessment  of  the  differences  between  the  estimates.  Big  differences 
should  trigger  further  investigations  on  the  possible  reasons  for  the  deviation.  In  particular,  to  enable  a  fair 
comparison  it  was  necessary  to  identify  any  costs  that  were  present  in  the  CAIG  model  but  not  in  the  GCBS 
model  and  vice  versa. 

7.2  Results 

The  final  results  are  shown  in  Figure  14: 

The  total  O&S  costs  computed  exclude  costs  associated  with  CAIG  cost  work  breakdown  elements  5.0 
(Continuing  System  Improvements)  and  6.0  (Indirect  Support).  The  main  reason  was  that  too  much  uncertainty 
existed  to  match  some  costs  to  these  aggregates.  Another  reason  is  the  total  absence  of  CAIG  5.0  in  the  GCBS 
model  and  CAIG  6.0  in  CAIG  estimates. 

7.3  Discussion 

Due  to  the  lack  of  data  and  information,  no  conclusions  could  be  drawn  on  the  accuracy  of  the  models.  There¬ 
fore,  the  observations  made  in  this  section  are  completely  based  on  a  qualitative  analysis  of  the  two  models: 

•  Aggregate  O&S  costs  in  the  GCBS-CAIG  model  are  not  directly  comparable  to  the  CAIG  model.  Al¬ 
though  the  CAIG  model  focuses  on  operations  and  support,  it  also  includes,  for  instance,  costs  for  con¬ 
tinuous  testing  and  development.  In  the  GCBS,  these  costs  belong  to  the  aggregate  “Production  phase”, 
which  is  not  part  of  the  O&S.  Furthermore,  the  CAIG  model  is  not  intended  for  crisis/war  scenarios, 
while  in  the  GCBS  model  (as  implemented  in  CATLOC)  a  war/crisis  scenario  could  easily  be  handled  by 
considering  it  as  a  mission  type. 

•  The  “personnel  costs”,  which  were  considered  to  be  overestimated  in  the  CAIG  model,  are  more  than 
30%  higher  in  the  GCBS  model.  However,  the  sum  of  CAIG  elements  1.0  and  2.0  together  seems  to  be 
close  to  the  GCBS  equivalent. 
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Cost  estimates  (ICE) 
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Figure  13:  Example  of  the  comparison  method 
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□  GCBS 

□  CAIG 


Total  O&S  CAIG  1.0  CAIG  2.0  CAIG  3.0  CAIG  4  0  CAIG  5.0  CAIG  6.0 
Costs 


Figure  14:  GCBS  vs.  CAIG 
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•  The  “maintenance  costs”  are  more  than  twice  as  high  in  the  GCBS  model.  If  we  consider  only  the  PM 
costs,  they  still  are  higher.  CAIG  element  4.0  should  be  excluded  from  the  comparison  because  there’s 
obviously  something  wrong  there. 

Without  more  data  it  is  not  possible  to  draw  any  conclusions  on  which  of  the  methods  produces  the  most 
reliable  cost  estimate.  Still,  even  if  it  has  not  been  possible  to  do  a  numerical  evaluation,  it  is  clear  that  the 
GCBS  model  is  more  relevant  than  the  CAIG  model  when  it  comes  to  understanding  the  origin  of  the  costs. 

If  the  only  objective  is  to  get  the  total  sum  of  all  costs,  a  thorough  investigation  may  show  that  the  two 
methods  are  more  or  less  equivalent.  If,  on  the  other  hand,  the  purpose  of  the  cost  analysis  is  to  find  cost  drivers, 
identify  opportunities  for  savings,  or  help  decision-makers  investment  decisions,  the  bottom-up  approach  should 
be  used. 

8.0  LESSONS  LEARNED 

The  section  is  divided  into  seven  parts:  the  general  management  of  the  estimating  process  and  the  feedback  of 
each  topic  of  the  cost  estimating  process;  aims  and  cost  boundary;  data  and  assumptions;  methods,  models  and 
tools;  risk  and  uncertainty;  analysis  of  the  results;  and  presentation  of  results. 

8.1  General  Management  of  the  Estimating  Process 

8.1.1  Initial  Plan 

SAS-076  started  with  a  plan  describing  the  activities  and  captured  this  in  a  Project  Initiation  Document  (PID). 
Originally  it  was  described  as  follows: 

For  the  acquisition  stage,  a  top-down  approach  would  be  adopted,  using  the  information  in  the  DADD  and 
applying  the  parametric  method.  As  a  second  estimating  method,  it  was  possible  to  apply  the  analogy  method 
using  public  available  data  (Internet,  NAO(UK)  reports,  etc.). 

Initially  the  cost-estimating  approach  consisted  of  two  iterations.  The  first  iteration  was  to  estimate  high- 
level  cost  elements  of  the  cost  breakdown  structure.  This  would  result  in  many  assumptions;  these  assumptions 
would  need  to  be  checked  by  the  RNLN.  It  would  then  have  been  used  to  identify  cost  drivers.  The  second 
iteration  was  to  go  into  more  details  for  the  costs  drivers,  if  additional  data  was  available. 

The  main  milestones  for  the  first  iteration  were: 

•  Milestone  1:  Agreement  on  cost  elements  to  be  calculated  -  16/01/2009. 

•  Milestone  2:  Release  of  the  data  and  assumptions  list  -  26/06/2009. 

•  Milestone  3:  Release  of  first  report  -  22/01/2010. 

The  main  Milestones  for  the  second  iteration  were: 

•  Milestone  4:  Agreement  on  new  cost  elements  to  be  calculated  -  22/01/2010. 

•  Milestone  5:  Release  of  the  updated  data  and  assumptions  list  -  09/07/2010. 

•  Milestone  6:  Release  of  second  iteration  report  -  01/10/2010. 
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8.1.2  Actual  Implementation  of  the  Initial  Plan 

The  following  steps  were  actually  followed  after  having  established  the  PID: 

•  Agreement  on  the  Cost  Breakdown  Structure  (CBS):  follow  ANEP  41  (SWBS)  and  the  CBS  from  SAS- 
054  report; 

•  Developed  a  Data  and  Assumptions  Document  Definition  (DADD)  based  on  the  data  found  in  public 
sources,  participant’s  inputs  and  the  RNLN; 

•  Investigated  software  tools  to  be  used. 

•  Decision  on  methods  to  be  use  for  O&S:  analogy,  parametric  and  engineering. 

Since  the  O&S  costs  for  a  materiel  system  are  in  high  extent  dependent  on  both  the  use  of  the  system  as 
well  as  the  systems  reliability  and  maintenance,  the  group  decided  to  apply  a  bottom-up/engineering  method 
to  estimate  the  O&S  costs.  The  group  proposed  to  use  the  type  of  estimating  approach  the  Swedish  Material 
Defence  Administration  has  experience  on.  This  approach  was  mainly  applied  to  evaluate  alternatives  in  acqui¬ 
sition,  maintenance  optimization,  improvement  of  the  in-service  cost  effectiveness,  etc.  Sweden  also  provided 
the  necessary  computer  tools  to  support  the  O&S  cost  estimate.  The  analysis  was  dependent  on  the  provision 
of  the  necessary  input  data  by  the  RNLN.  Since  both  the  Rotterdam  and  the  Johan  de  Witt  have  been  used  and 
maintained  for  quite  some  years,  the  availability  of  data  was  expected  to  be  high. 

8.1.3  Lessons  Identified 

•  The  group  did  not  work  continuously  on  this  project,  causing  additional  challenges  in  the  estimates  and 
in  the  management  of  some  important  issues. 

•  The  SAS-076  group  considered  the  estimating  guideline  and  the  tasks  described  in  SAS-054  of  the  man¬ 
agement  process  to  be  useful. 

•  The  group  tried  to  follow  the  SAS-054  guidelines  as  much  as  possible,  however  the  group  was  restricted 
by  the  limited  availability  of  resources  and  competence  available  in  the  group  or  available  within  the  group 
members’  professional  network.  This  affected,  to  a  certain  extent,  the  work  process.  The  conclusion 
was  that  the  SAS-054  guidelines  could  be  followed,  however,  it  takes  time  and  resources.  Therefore, 
the  resources  and  the  budget  required  to  do  an  estimate  according  to  the  SAS-054  guideline  should  be 
carefully  planned  before  the  actual  work  starts. 

8.2  Definition  of  the  Aims  and  Cost  Boundary 

8.2.1  The  Team’s  Approach 

The  customer  for  the  results  of  the  study  was  the  RNLN.  As  this  estimate  is  also  used  to  apply  the  guideline 
developed  by  SAS-054,  NATO  RTO  can  be  considered  as  a  secondary  customer. 

The  aim  was  to  practice  the  guidelines  of  SAS-054.  The  objective  for  the  O&S  costs  estimation  was  to 
approximate  the  yearly  in-service  support  costs  and  to  sum  them  for  a  chosen  life-time  given  the  planned 
operational  profile.  This  would  result  in  finding  the  major  cost  drivers  and  eventually  give  the  RNLN  some 
opportunities  to  improve  their  maintenance  plans. 
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For  the  O&S  costs,  the  cost  boundary  changed  during  the  cost  estimating  process,  e.g.,  helicopters  were 
first  not  included.  The  Rotterdam  group  included  the  costs  for  the  helicopters  as  the  RNLN  informed  the  group 
that  these  costs  were  included  in  the  data.  Originally  the  disposal  costs  were  included  in  the  Cost  Breakdown 
Structure,  but  the  RNLN  could  not  provide  a  strategy  for  the  disposal  of  the  ships,  which  resulted  in  the  removal 
of  these  costs  from  the  Cost  Breakdown  Structure. 


8.2.2  Lessons  Identified 

•  The  aim  and  the  objective  of  the  study  are  fundamental  for  the  way  the  cost  analysis  is  conducted. 

•  For  this  ICE  the  generic  Cost  Breakdown  Structure,  as  defined  in  the  SAS-054  Report,  was  used  and 
considered  to  be  useful.  The  cost  elements  were  estimated  were  defined  iteratively. 

8.3  Data  and  Assumptions 

8.3.1  The  Team’s  Approach 

It  was  decided  to  use  three  approaches  to  estimate  the  O&S  costs:  parametric,  analogy  approach  (CAIG  esti¬ 
mate),  and  the  engineering  bottom-up  approach. 

A  first  estimate  for  the  O&S  costs  was  made  using  O&S  cost  data  retrieved  from  VAMOSC  (Visibility  and 
Management  of  Operating  &  Support  Costs)  data  base.  The  CBS  followed  OSD  Cost  Analysis  Improvement 
Group  (CAIG)  Cost  Element  Structure,  which  provides  a  standardized  system  to  prepare  and  submit  O&S 
cost  estimates.  Analogous  ships  were  selected  on  the  basis  of:  dimensions,  ship  technical  characteristics,  and 
intended  role/mission.  Data  was  then  averaged  over  the  life-time  of  the  ship,  and  the  data  points  were  used  as 
a  basis  for  the  cost  estimating  relationships  (CERs).  Two  estimating  methods  were  used  to  compute  the  CAIG 
elements:  a  parametric  approach  -  using  only  one  parameter  for  each  CAIG  element,  and  an  analogy  method 
using  LSD-49  Class  averages.  The  outcome  from  both  methodologies  was  then  combined,  leading  to  an  initial 
cost  estimate  for  the  O&S  costs. 

The  engineering  approach  to  estimate  the  O&S  cost  is  to  a  great  extent  based  on  the  connection  between 
Reliability,  Availability,  Maintainability  and  Supportability  data  (RAMS),  and  costs  for  spare  parts,  man-hours 
and  other  cost  elements.  The  costs  are  computed  in  the  cost  atoms  algorithms  in  the  cost  model.  For  this,  it  is 
important  to  have  detailed  knowledge  of  the  technical  and  organizational  data.  Since  the  group  could  not  obtain 
the  required  reliability  data  to  estimate  the  costs  for  corrective  maintenance  (mainly  failure  rates),  it  was  not 
possible  to  calculate  the  costs  according  to  the  engineering  method.  This  is  of  great  importance  and  had  a  severe 
impact  on  accuracy  of  the  cost  estimate  since  the  Corrective  Maintenance  (CM)  is  often  a  major  cost  driver  in 
the  in-service  phase  of  a  system.  As  a  result,  a  combination  of  CAIG  data  and  Swedish  experience  with  respect 
to  the  fraction  of  CM  cost  related  to  the  total  O&S  cost  was  used  instead.  Naturally  the  purpose  was  to  have  a 
full  estimate  of  the  O&S  cost,  but  data  availability  to  apply  the  engineering  method  was  insufficient 

For  this  O&S  estimate,  the  generic  Cost  Breakdown  Structure,  as  defined  in  the  SAS-054  Report,  was 
used.  The  cost  elements  that  were  estimated  were  defined  iteratively.  The  methodology  defined  in  the  SAS-028 
Report  with  the  generic  Resources,  Product  and  Task  dimensions  was  used.  Also,  the  definition  of  the  generic 
System  Breakdown  Structure,  ANEP41,  was  used.  If  necessary,  these  approaches  were  tailored  according  to 
the  specifics  of  the  program.  For  this  assessment,  the  O&S  costs  were  estimated  using  both  organizational  and 
operational  data,  as  well  as  technical  and  financial  data.  Data  was  gathered  by  the  RNLN.  A  more  detailed 
description  of  the  data  can  be  found  in  the  Section  4  and  Appendix  1  of  this  Annex. 
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ORGANIZATION 


Regarding  the  cost  normalization  for  the  parametric  and  analogy  methods  used  in  CAIG  estimate,  the  data 
was  taken  from  the  VAMOSC  database  with  all  costs  in  constant  fiscal  year  (FY)  or  base  year  2009  U.S.  Dollars. 
In  order  to  deal  with  the  high  variability  in  O&S  costs,  a  two  step  approach  was  undertaken: 

•  Data  from  VAMOSC  was  averaged  on  an  annual  basis  between  different  ships  in  each  class; 

•  Then  the  yearly  data  for  each  class  of  ship  was  averaged  over  the  life-time  of  the  ships  in  all  cases 
beginning  in  1984  and  ending  when  the  ship  was  either  decommissioned  or  at  the  end  of  the  sample 
period. 

The  costs  for  the  engineering  method,  provided  by  the  RNLN  in  euros  base  year  2005,  were  normalized 
using  the  eurostat  labour  index  and  converted  to  euros  base  year  2008. 

8.3.2  Lessons  Identified 

•  Lessons  learned  related  to  the  CAIG  estimate: 

-  Using  a  single  parameter  to  estimate  costs  may  not  produce  reliable  estimates.  For  example  CAIG 
elements  3  &  4  are  computed  on  a  more  or  less  accurate  relationship  between  displacement  and 
maintenance/sustainment  costs.  The  maintenance  methods  and  schedules  should  also  be  taken  into 
account  here. 

-  CER  (cost  estimating  relationships)  with  FLD  (Full  load  Displacement)  only  works  when  ships 
being  compared  have  similar  manning,  operational  and  maintenance  philosophies. 

-  The  best  way  to  predict  fuel  consumption  is  to  have  specific  data  on  steaming  hours,  fuel  costs, 
engine  efficiency  and  the  units’  usage  portfolio. 

-  System  complexity,  number  of  systems  and  many  other  factors  can  all  have  an  impact  on  costs. 

-  Databases  like  VAMOSC  are  very  useful  to  estimate  O&S  costs. 

•  Lessons  learned  related  to  the  engineering  method: 

-  Access  to  Follow-Up  data  from  the  In-Service  phase  was  a  necessary  condition  to  be  able  to  apply 
the  engineering  method  for  estimating  O&S  costs.  If  that  condition  could  not  be  fulfilled,  rough 
assumptions  would  have  to  be  made.  An  alternative,  not  performed  for  the  current  estimation,  was 
to  use  parametric  COTS  models  to  complete  the  estimate. 

-  One  has  to  bear  in  mind  that  the  engineering  method  allows  you  not  only  to  calculate  the  costs 
themselves,  but  also  the  costs  related  to  the  specific  system  and  its  operational  profile  and  support 
organization.  The  engineering  method  allows  the  user  to  vary  the  above  mentioned  data  and  to 
calculate  the  costs  for  this  variation.  Thus,  the  engineering  method  allows  the  user  to  determine  the 
cost  efficiency  of  the  system.  This  implies  that  the  method  is  considered  to  be  both  expensive  and 
vulnerable.  The  latter  implies  its  dependence  of  accurate  Follow-Up  and  RAMS  data.  This  has  been 
exemplified  by  the  fact  that  the  group  was  not  able  to  gather  all  the  necessary  data. 

8.4  Methods,  Models  and  Tools 

8.4.1  The  Team’s  Approach 

Three  methods  were  used:  an  Engineering  bottom-up,  a  parametric,  and  an  analogy  approach.  However,  it 
appeared  that  the  engineering  bottom-up  was  not  applicable  as  such  due  to  the  lack  of  engineering  data  such  as 
reliability  of  the  items.  The  engineering  method  used  was  rather  a  mix  of  top-down  and  bottom-up  methods. 
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The  tools  investigated  were  mainly  commercial.  It  was  not  possible  to  acquire  one  of  the  tools,  due  to  the 
very  limited  resources  available.  Therefore  the  group  used  tools  that  were  already  available  in  one  or  more 
nations  and  for  which  a  temporary  licence  was  given  to  the  participating  nations  (CATLOC). 

8.4.2  Lessons  Identified 

•  The  data  available  determined,  to  a  very  large  extent,  the  choice  of  methods,  from  the  start  of  the  cost 
analysis  process  to  the  end  of  it.  It  was  a  continuous  interaction  between  choices  of  methods,  computer 
models  and  availability  of  data.  For  example,  the  methodologies  were  chosen  assuming  that  certain  data 
would  be  available,  however,  some  data  was  not  available  and  forced  the  team  to  adapt  the  methodology, 
or  to  gather  the  data  from  a  different  source. 

•  The  SAS-054  report  discusses  the  engineering  method  (bottom-up).  The  description  of  the  engineering 
method  is  correct,  however,  reliability  data  should  be  mentioned  explicitly.  Suggestions  include  adapting 
the  following  sentence:  “This  type  of  estimate  is  used  when  detailed  design  data,  such  as  reliability  data 
on  the  system,  is  available”. 

•  Furthermore,  the  engineering  method  is  not  the  same  as  a  bottom-up  approach.  Flandbooks  (including 
SAS-054)  mix-up  approaches  (top-down  and  bottom-up),  and  methodologies  (parametric,  engineering). 
Top-down  and  bottom-up  are  dependent  on  the  granularity  of  the  Cost  Breakdown  Structure  and  the 
available  data.  What  is  missing  in  the  SAS-054  report  is  the  specification  of  the  top-down  and  the  bottom 
up  approaches,  before  detailing  the  types  of  methodologies. 

8.5  Risk  and  Uncertainty  Analysis 

8.5.1  The  Team’s  Approach 

Most  data  was  received  from  the  RNLN  and  was  considered  reasonably  accurate.  However,  some  parameters 
collected  outside  of  the  RNLN  have  uncertainties  attached  to  them.  These  are:  fuel  price,  labour  cost  escalation 
and  the  cost  elements  for  which  it  was  not  possible  to  obtain  data. 

8.5.2  Lessons  Identified 

When  applying  an  engineering  method,  it  is  inevitable  that  some  data  are  uncertain.  As  a  result,  there  is  a  need 
to  adjust  collected  data  to  the  LCC  model  for  the  analysis  in  question.  The  way  to  handle  these  uncertainties  is 
to  conduct  a  sensitivity  analysis  or  to  apply  Monte  Carlo  simulation.  For  this  study,  this  was  not  done. 

8.6  Analysis  of  Results 

8.6.1  The  Team’s  Approach 

The  team  showed  the  results  for  both  the  CAIG  CBS  and  the  GCBS.  The  engineering  method  (GCBS)  allows 
showing  detailed  results  and  the  distribution  of  costs  among  the  cost  aggregates  and  cost  atoms.  This  way  cost 
drivers  can  be  identified  and  more  detailed  analysis  on  costs  can  be  started.  However,  because  of  the  lack  of 
data,  this  type  of  analysis  could  not  be  performed. 
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8.6.2  Lessons  Identified 

Although  sufficient  data  was  not  available  to  perform  a  detailed  analysis  on  the  costs,  it  was  useful  to  present 
the  results  as  detailed  as  possible.  This  way,  areas  can  be  identified  for  further  detailed  analysis.  Furthermore, 
the  CATLOC  model  proved  to  be  very  useful  when  applying  the  engineering  method  and  provided  very  useful 
graphical  outputs. 

Another  lesson  is  that  showing  the  features  of  a  model  can  be  used  as  a  catalyst  for  further  data  collection. 
Once  customers  are  aware  of  what  type  of  analysis  is  possible  with  a  certain  model,  they  are  more  than  willing 
to  gather  more  detailed  data. 

8.7  Presentation  of  the  Results 

8.7.1  The  Team’s  Approach 

Intermediate  results  were  shown  to  the  RNLN  with  the  intention  of  persuading  them  to  gather  more  data.  This 
was  done  by  explaining  why  the  extra  data  was  required  and  what  was  possible  if  data  was  to  be  provided. 
This  was  part  of  the  “learning  process”  included  in  the  methodology.  Both  the  cost  estimator  and  the  operator 
of  the  system  were  learning  from  each  other.  The  first  one  was  learning  what  the  user  wants  how  the  system 
was  operated  and  structured  and  the  second  learns  why  it  was  necessary  to  capture  costs,  what  the  relation  was 
between  the  RAMS  data  and  the  costs  and  finally  how  he  could  improve  the  effectiveness  of  his  system. 

8.7.2  Lessons  Identified 

•  The  SAS-076  task  group  did  unintentionally  assume  that  the  data  provided  by  the  RNLN  was  coherent 
with  the  CBS,  which  was  not  the  case.  This  could  be  regarded  as  a  typical  mistake  of  an  international 
study  group  or  an  example  of  how  input  data  determines  the  method.  Therefore,  during  the  acquisition 
phase  of  a  system,  the  prerequisites  for  gathering  data  necessary  to  the  future  cost  estimations  should  be 
established.  This  means  that  follow-up  databases  and  input  data  structures  should  be  implemented  very 
early. 

•  Even  if  the  group  could  not  obtain  the  data  necessary  to  properly  apply  the  engineering  method,  and  did 
not  get  the  desired  outcomes,  the  method  worked.  The  learning  process  discussed  above  instigated  a  shaip 
reaction  within  the  RNLN.  They  have  already  decided  to  improve  the  ways  they  collect  data  for  future 
cost  estimates. 

9.0  CONCLUSIONS 

The  Rotterdam  group  managed  to  produce  an  estimate  for  the  O&S  costs  based  on  the  planned  operational 
profile.  Success  depended  on  the  availability  of  the  right  expertise,  the  right  computer  tools,  and  the  right  data 
sources.  For  the  latter,  especially  for  O&S  cost  estimation,  open  sources  were  helpful  but  any  estimator  cannot 
solely  rely  on  them. 

The  engineering  method  used  to  estimate  cost  in  a  budgeting  process,  especially  from  a  historical  perspec¬ 
tive,  was  not  an  optimal  approach.  Concerning  budgeting  and  financing  it  is  often  required  that  all  the  costs 
should  be  presented,  and  for  that  more  appropriate  economical  methods  already  exist.  However,  the  engineer¬ 
ing  method  is  a  very  powerful  tool,  if  not  the  only  one,  to  identify  and  reduce  cost  drivers  during  the  critical 
decision  making  in  the  systems  lifetime. 
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The  engineering  method  should  concentrate  on  the  objective  of  the  analysis  the  user  and  the  cost  estimator 
would  like  to  achieve.  The  boundaries  for  work,  type  of  data,  amount  of  data,  the  design  of  the  CBS  etc.,  differ 
according  to  the  type  of  analysis.  For  example,  not  all  the  costs  or  RAMS  parameters  influence  the  analysis  to 
be  used.  The  engineering  method  is  trying  to  minimize  the  need  of  input  data  taking  into  consideration  only 
those  costs  and  related  parameters  which  influence  the  decision  making  process. 

It  is  probably  better  to  look  at  the  engineering  method  as  an  on-going  enterprise  during  the  system  life 
time,  i.e.,  “from  cradle  to  grave”.  Applied  properly  and  consistently,  the  method  not  only  implicitly  leads  to 
improvement  of  the  system  efficiency,  but  also  gives  the  system  operator  after  a  period  of  time,  access  to  a 
database  similar  to  VAMOSC  which  will  substantially  improve  the  future  Life  Cycle  Cost  estimations. 

The  cost  estimate  resulting  from  this  study  should  be  considered  as  indicative.  All  the  uncertainties  related 
to  the  operating  profiles  during  the  system  lifetime,  the  reliability,  maintainability  and  supportability  issues,  the 
impacts  of  future  technology,  and  the  economical  fluctuation  on  the  market  in  general  are  reasons  to  believe 
that  an  accurate  estimate  of  the  life  cycle  costs  is  unfeasible.  However,  alternative  solutions  can  usually  be 
evaluated  with  higher  degrees  of  consistency  by  certifying  that  the  levels  of  uncertainty  are  similar  for  all 
considered  alternatives.  The  engineering  method  may  come  closest  to  reality. 
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Appendix  B2-1:  CATLOC  LCC  MODEL  DESCRIPTION 

Figure  15  shows  how  the  LCC-tree  can  be  viewed  in  CATLOC. 


I:\FMV\SAFEHD\Uppgifter_Logmodeller\l  Metodstod\SAS  076  medverkan\Arbetslage\GCBS_ver3. 1  .cci 


e  [CD  gcbs 

ft  ■  LCC,  Life  Cycle  Cost 
9  I  7,  In  Service  Phase 
ft  H  7.1,  Operation 

a  H  7.1.1,  Personnel  costs 

*  |  7.1.2,  Petroleum.  Oil  and  Lubricants 
a  ■  7.1.3,  Munitions'  Costs 

*  ■  7.1.4,  Cost  of  other  consumables  (food,  water,  soft  drinks,  etc) 
ft  ■  7.2,  Supporting  operations 

a  □  7.2.1,  Costs  of  updating  operating  documentation, 
a  □  7.2.4,  Cost  of  embedded  softwares 

a  n  7.2.3,  Costs  updating  computer  software/hardware  for  information  systems, 
a  □  7.2.2,  Costs  of  use  infrastructure  for  operating  the  system 
ft  ■  7.3,  Maintenance 

a  ■  7.3.3,  Costs  related  with  other  operating  systems 
a  ■  7.3.1,  Cost  of  Preventive  Maintenance 
a  |  7.3.2,  Cost  of  Corrective  Maintenance 
a  □  7.3.4,  Cost  of  test  and  support  equipment  and  tools 
a  □  7.3.5,  Cost  for  updating  maintenance  documentation 
□  7.3.7,  Costs  for  contract  logistic  support 
a  □  7.3.6,  Cost  of  yearly  replenishment  of  spares 
ft  ■  7.5,  PHST 

aD  7.5.1,  Materiel  costs  for  handling  of  Government  Owned  Stores 
a  □  7.5.2,  MHC  for  handling  of  Government  Owned  Stores 
ft  ■  7.4,  Sustatining  Support 

a  □  7.4.1,  Costs  of  periodic  training  operating  personnel 
a  □  7.4.4,  Yearly  acquisitions  cost  for  trainings  devices 
a  □  7.4.3,  Yearly  cost  for  training  devices 
a  □  7.4.2,  Costs  of  periodic  training  maintenance  personnel 
a  ■  7.6,  Restoration 
ft  ■  7.7,  Indirect  Support 

a  □  7.7.2,  Materiel  costs  generated  by  Navy  Command  ( Headquarters,  Personnel  &  Operations ) 
an  7.7.1,  Man  hour  costs  generated  by  Navy  Command  ( Headquarters,  Personnel  &  Operations ) 

a  Mi,  lcca 

■  1.7,  IEAC 
a  0  Default  View 


Figure  15:  The  LCC  tree  as  listed  in  the  CATLOC  model 

The  CATLOC  model  shown  in  Table  3  below  is  based  upon  a  template  implementing  a  generic  cost  break¬ 
down  structure  (GCBS)  proposed  by  the  NATO  RTO  Task  Group  SAS-054.  The  generic  hierarchy  is  described 
in  section  10.3  and  12. 1  of  the  SAS-054  final  report.  The  template  includes  cost  aggregates  and  a  cost  structure 
but  no  formulas,  cost  atoms  or  domain  dimensions. 
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The  model  was  populated  with  cost  atoms  according  to  the  initial  assumptions  made  by  SAS-076,  and  after 
that  it  was  adjusted  and  tailored  using  the  data  obtained  from  the  RNLN.  Some  cost  aggregates,  or  cost  atoms, 
have  been  kept  in  the  model  even  if  input  data  was  not  available.  The  main  reason  for  this  was  to  illustrate 
the  benefits  the  engineering  LCC  methodology  has  if  data  was  available.  During  a  demonstration  session  these 
benefits  were  shown  to  the  RNLN.  The  RNLN  decided  to  have  a  closer  look  at  the  model  and  eventually  decided 
to  obtain  the  necessary  data  to  perform  a  complete  analysis. 

The  cost  aggregates  in  Tables  3-5  comprise  the  sum  of  the  cost  aggregates  or  cost  atoms  from  the  next 
lower  level.  A  cost  atom  is  usually  described  by  a  parameter  or  a  formula.  For  a  better  understanding  of  the 
model,  first  a  “Level  report”  (Tables  3  -  5)  is  shown,  representing  the  relation  between  cost  aggregates  and  cost 
atoms.  Then  a  “Tree  report”  (Tables  6-10)  is  shown,  revealing  the  description  of  the  parameters  used  in  the 
formulae.  In  both  tables,  the  Title/Index  column  gives  the  name  of  the  LCC  tree  node.  In  the  Level  report  only, 
the  cost  aggregates  have  a  formula  (a  sum)  on  the  row  below  the  description. 


Table  3:  The  CATLOC  model  A1-1  Level  reporting  table  (Levels  0-2) 


Level 

Title/Index 

Description 

LEVEL  0 

LCC 

Life  Cycle  Cost 

7+  1 

LEVEL  1 

7 

In  Service  Phase 

7.1  +  7.2  +  7.3  +  7.5  +  7.4  +  7.6  +  7.7 

1 

LCCA 

1.7 

LEVEL  2 

7.1 

Operation 

7.1.1  +7.1.2 +  7.1.3 +  7.1.4 

7.2 

Supporting  operations 

7.2.1  +7.2.2 +  7.2.3 +  7.2.4 

7.3 

Maintenance 

7.3.1  +  7.3.2  +  7.3.3  +  7.3.5  +  7.3.4  +  7.3.6  +  7.3.7 

7.4 

Sustaining  Support 

7.4.1  +7.4.2 +  7.4.3 +  7.4.4 

7.5 

PHST 

7.5.2 +  7.5.1 

7.6 

Restoration 

7.6.1 

7.7 

Indirect  Support 

7.7.1  +7.7.2 

1.7 

IEAC 
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Table  4:  The  CATLOC  model  A1-1  Level  reporting  table  (Level  3) 

Level  Title/Index 

Description 

LEVEL  3 

7.1.1 

Personnel  costs 

7.1.2 

7. 1.1.1  +  7. 1.1.2 

Petroleum,  Oil  and  Lubricants 

7.1.3 

7. 1.2.1  +  7. 1.2.2  +  7. 1.2.3  +  7. 1.2.4  +  7. 1.2.5 

Munitions’  Costs 

7.1.4 

7. 1.3.1 

Cost  of  other  consumables  (food,  water,  soft  drinks,  etc) 

7.2.1 

7. 1.4.1  +  7. 1.4.2  +  7. 1.4.3  +  7. 1.4.4 

Costs  of  updating  operating  documentation 

7.2.2 

Costs  of  use  infrastructure  for  operating  the  system 

7.2.3 

Costs  updating  computer  software/hardware  for  information  systems 

7.2.4 

Cost  of  embedded  software 

7.3.1 

Cost  of  Preventive  Maintenance 

7.3.2 

7.3. 1.1  +  7.3. 1.2  +  7.3. 1.3  +  7.3. 1.4  +  7.3. 1.5 

Cost  of  Corrective  Maintenance 

7.3.3 

7.3.2. 1  +  13.2.2  +  13.23 

Costs  related  to  other  operating  systems 

7.3.5 

7.3.3. 1  +  7.3. 3.2 

Cost  for  updating  maintenance  documentation 

7.3.4 

Cost  of  test  and  support  equipment  and  tools 

7.3.6 

Cost  of  yearly  replenishment  of  spares 

7.3.7 

Costs  for  contract  logistic  support 

7.5.2 

Man-Hour  cost  for  handling  of  Government  Owned  Stores 

7.5.1 

Materiel  costs  for  handling  of  Government  Owned  Stores 

7.4.1 

Costs  of  periodic  training  operating  personnel 

7.4.2 

Costs  of  periodic  training  maintenance  personnel 

7.4.3 

Yearly  cost  for  training  devices 

7.4.4 

Yearly  acquisitions  cost  for  trainings  devices 

7.6.1 

Cost  of  recommended  and  mandatory  modifications  of  the  system 

7.7.1 

Man  hour  costs  generated  by  Navy  Command  (Headquarters,  Personnel 

7.7.2 

Operations) 

Materiel  costs  generated  by  Navy  Command  (  Headquarters,  Personnel  &  Operations) 
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Table  5:  The  CATLOC  model  A1-1  Level  reporting  table  (Levels  4-5) 

Level  Title/Index  Description 
LEVEL  4 

7 . 1. 1. 1  Costs  of  operating  personnel 

7. 1. 1.2  Average  personnel  costs  for  operation  of  helicopters 

7. 1.2.1  Cost  of  fuel 

7.1 .2.2  Cost  of  oil  consumption 

7. 1.2.3  Cost  of  lubricants 

7.1 .2.4  Yearly  costs  for  POL,  Ship 

7.1 .2.5  Average  POL  costs  due  to  maintenance  of  helicopter  per  year 

7. 1.3. 1  Costs  of  munitions  and  countermeasures  consumption 

7. 1.4. 1  Average  cost  for  food  per  year 

7.1 .4.2  Average  yearly  cost  for  water  consumption 

7. 1.4.3  Average  yearly  cost  for  electricity  consumption 

7.1 .4.4  Average  yearly  cost  of  "other  operational  material" 

7.3. 1. 1  Annual  man  hour  costs  for  major  preventive  maintenance  action  on  main  system 

7.3. 1.2  Materiel  costs  for  major  preventive  maintenance  action  on  main  system 

7.3. 1.3  CHPMR,  Annual  man  hour  costs  for  PM  actions  on  unit  level 

7.3.1 .4  CMPMT,  Annual  materiel  costs  for  PM  actions  on  unit  level 

7.3. 1.5  CPMTC,  Annual  costs  for  PM  actions  on  unit  level  performed  by  contractor 

7.3.2. 1  Average  CM  costs  due  to  unit  replacement  actions 

7. 3. 2. 1. 1+7. 3. 2. 1.2+7.3. 2. 1.3+7. 3. 2. 1.4+7. 3. 2. 1.7+7. 3. 2. 1.8  +7.3.2. 1.5+7.3.2. 1.6 

7 .3.2.2  Average  CM  costs  due  to  unit  repair  actions 

7. 3. 2.2.1  +  13.2.2.2  +  13.2.23  +  7.3.2.2.4  +  13.2.2.5 

13.23  Average  yearly  CM  cost ,  both  material  and  man  power 

7.3.3. 1  Average  man-hour  cost  for  maintenance  of  helicopters  (maintenance  staff) 

1 33.2  Average  materiel  costs  due  to  maintenance  of  helicopter  per  year 

LEVEL  5 

7.3.2. 1 . 1  CHCMRP.  Man  hour  costs  for  unit  replacements  due  to  CM  during  operational  time 
onboard  the  ship  performed  by  NME  at  OML  level 

7.3.2. 1.2  CMCMRP,  Average  cost  for  materiel  consumption  (consumables)  due  to  CM 
replacement  actions  at  OML  performed  by  the  crew 

7.3.2. 1.3  CTCMRP,  Average  cost  for  transportation  due  to  CM  replacement  action  at  OLM 
performed  by  NME  personnel  abroad 

7.3.2. 1 .4  CCMRPEX,  Average  cost  for  CM  replacements  at  unit  level  performed  by  external  personnel 

7. 3. 2. 1.7  TRCCMRP,  CMRp  -  Average  transportation  costs  for  replacements  actions  due  to  CM  at  OLM  performed 
by  personnel  from  NME  per  year  -  includes  personnel  and  replaced  unit  transportation  back  and  forth 

7.3.2. 1 .8  CMRPEX,  CMRp  -  Average  cost  for  replacement  actions  if  performed  by  contractor  /  external  man-power  at  OLM 

7.3.2. 1.5  CMRPCC,  CMRp  -  Average  cost  of  consumables  consumption  for  replacements  due  to  CM  at  OLM  per  year 

7.3.2. 1 .6  MHCCMRP.  CMRp  -  Average  man-hour  cots  for  replacements  due  to  CM  at  OLM 
performed  by  personnel  from  NME  per  year 

13.2.2. 1  CCMRNME,  Average  CM  costs  for  repair  actions  at  unit  level  performed  by  NME 
per  year 

1 3. 2.2.2  CMCMCT,  Average  CM  costs  for  repair  actions  at  unit  level  performed  by  contractor  per  year 

1 3.2.23  CMCMCC,  CM  -  Average  cost  of  consumables  for  repair  actions  due  to  CM  per  year 

7. 3. 2. 2.4  MHCCMR,  CM  -  Average  man-hour  cots  for  repair  actions  due  to  CM  per  year 

13.2.2.5  CMCMEX,  CM  -  Average  cost  for  repair  actions  due  to  CM  per  year  performed  by  contractor 
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In  Tables  6-10  in  the  Description  column,  the  first  row  corresponds  with  the  title  of  the  node.  The  next  row 
provides  the  formula  used,  if  applicable.  Then  the  parameter  used  is  described.  In  some  cases  the  parameter 
used  is  the  result  of  a  calculation.  In  this  case  the  formula  is  shown  below  the  parameter  description.  Finally 
the  row  before  the  next  node  is  reserved  for  remarks.  The  sum  for  cost  aggregates  is  no  longer  shown. 


Table  6:  The  CATLOC  model  A1-2  Tree  reporting  table 


Title/Index 

Description 

7 

In  Service  Phase 

7.1 

Operation 

7.1.1 

Personnel  costs 

7.1.1. 1 

Costs  of  operating  personnel 

NOPT*(MIOP+COPC) 

NTOPP  -  Number  of  different  types  of  operational  personnel.  Not  used  explicitly  in  the  model 

NOPT  -  Number  of  operators  per  type  (manning  profile) 

MIOP  -  Average  yearly  income  of  operators  (salaries  +  benefits) 

COPC  -  Average  yearly  cost  of  on-board  personnel  uniforms,  accessories,  shoes,  special 
clothing,  etc 

7.1. 1.2 

Average  personnel  costs  for  operation  of  helicopters 

NH  ALO  *  CPOPH  Y 

NHALO  -  Number  of  Helicopters  on  the  ship 

CPOPHY  -  Average  personnel  costs  for  operation  of  helicopters 

7.1.2 

Petroleum,  Oil  and  Lubricants 

7. 1.2.1 

Cost  of  fuel 

CLFS*MFCH*UTILIS*8.760 

NFT  -  Number  of  types  of  fuel  consumed  per  system  per  mission.  The  model  allows  2  types  of  fuel 

MFCH  -  Average  fuel  consumption/operational  hour/type  of  fuel  in  litres  (1) 

IDFUEL  -  Unique  identifier  for  fuel  type 

IDMISSION  -  Unique  identifier  for  mission  type 

IMFCH  -  RNLN,  input  data  ,  fuel  consumption  per  year 

CLFS  -  Average  cost  for  fuel  per  type  of  fuel  in  1000*EUR/1 

RNLN  input 

7. 1.2. 2 

Cost  of  oil  consumption 

MOCH*CLOS*UTILIS*8.760 

NOT  -  Number  of  types  of  oil  used  per  system  per  mission 

MOCH  -  Average  oil  consumption  per  operational  hour  per  type  of  oil 

An  analogous  formula  as  for  the  fuel  consumption  is  used  for  the  oil  consumption.  However, 
since  these  cost  atoms  are  not  used,  the  formula  and  its  parameters  are  not  presented  here. 

CLOS  -  Average  cost  for  oil  consumption  per  operational  hour  per  type  of  oil  used 

NOT  is  defined  in  CATLOC  explicitly  in  the  materiel  domain,  ALL_OIL_TYPES  which  contains 

for  the  moment  2  types  of  lubricants  OIL1  &  OIL2  No  costs  are  calculated  here,  as  the  oil  consumption  is  unknown 

7. 1.2. 3 

Cost  of  lubricants 

MLCH*CLGS*UTILIS*8760 

NLT  -  Number  of  types  of  lubricants  used  per  system  per  mission 

MLCH  -  Average  lubricant  consumption  per  operational  hour  per  type  of  lubricants 

Analogous  formula  as  for  fuel  consumption  is  used  for  the  lubricants  consumption  but  since 
these  cost  atoms  are  not  used,  the  formula  and  its  parameters  are  not  presented  here. 

CLGS  -  Average  cost  for  lubricants  consumption  per  operational  hour  per  type  of  lubricants 

NLT  is  defined  in  CATLOC  explicitly  in  the  materiel  domain,  ALL_LUBRICANT_TYPES  which  contains  for  the  moment 

2  types  of  lubricants  LUBRICANT  1  &  LUBRICANT2.  No  costs  are  calculated  here,  as  the  lubricant  consumption  is  unknown. 

7. 1.2. 4 

Yearly  costs  for  POL,  Ship 

CPOLSH 

CPOLSH  -  Average  cost  for  POL,  ship 

In  this  case  the  formula  is  just  an  input  parameter. 

RNLN  input 

7. 1.2. 5 

Average  POL  costs  due  to  maintenance  of  helicopter  per  year 

NHALO  *  CPOLH  Y 

CPOLHY  -  Average  POL  costs  due  to  maintenance  of  helicopter  per  year 

RNLN  input 

RTO-TR-SAS-076 


B2  -41 


ANNEX  B2  -  ESTIMATING  THE  OPERATION  AND 
SUPPORT  COSTS  OF  THE  ROYAL  NETHERLANDS  NAVY 
LANDING  PLATFORM  DOCK  SHIPS:  AN  EX  ANTE  ANALYSIS 


ORGANIZATION 


Table  7:  The  CATLOC  model  A1-2  Tree  reporting  table  (Cont’d) 

Title/Index  Description 

7.1.3  Ammunition  Costs 

7. 1.3.1  Costs  of  ammunition  and  countermeasures  consumption 

MMCY*CMUY 

MMCY  -  Average  units  of  ammunition  type  i  consumed  per  year 

CMUY  -  Average  cost  per  unit  of  munitions  type  i 

More  than  one  type  of  ammunition  can  be  defined  (RNLN  input) 

7.1.4  Cost  of  other  consumables  (food,  water,  soft  drinks,  etc) 

7. 1 .4. 1  Average  cost  for  food  per  year 
NMOB  *CFOB  *UTILF*365 

NMOB  -  Number  of  man  on-board  the  ship 
CFOB  -  Average  cost  of  food  per  person  per  day 
Assumed  cost,  see  section  4 

7. 1 .4.2  Average  yearly  cost  for  water  consumption 
CWCU*MWCY 

CWCU  -  Average  cost  per  water  unit 
MWCY  -  Average  water  consumption  per  year 
Assumed  cost.  See  section  4 

7. 1 .4.3  Average  yearly  cost  for  electricity  consumption 
CECU*MECY 

CECU  -  Electricity  cost  per  kWh 

MECY  -  Average  electricity  consumption  per  year 

No  data  available,  not  used. 

7. 1 .4.4  Average  yearly  cost  of  “other  operational  material”  CCOE 

CCOE  -  Yearly  cost  of  “other  operational  material”  No  data  available,  not  used. 

7.2  Supporting  operations 

7.2.1  Costs  of  updating  operating  documentation. 

NDOU*NDOPY*CDOU 

CDOU  -  Cost  per  operating  documentation  update  of  type  i  per  year 
NDOU  -  Number  of  operating  documentation  updates  for  each  type  per  year. 

NDOPY  -  Number  of  operating  documentation  updates  of  type  i  per  year 

Only  CDOU  has  an  assumed  input  defined  in  section  4  representing  the  total  cost  per  year  for  documentation  updates. 

7.2.2  Costs  of  use  infrastructure  for  operating  the  system 
NDFH*CDFH 

NDFH  -  Period  in  harbours  per  year 
CDFH  -  Average  fee  in  harbour  per  day 
Not  enough  data  available,  not  used 

7.2.3  Costs  updating  computer  software/hardware  for  information  systems. 

No  data  available,  not  used 

7.2.4  Cost  of  embedded  software 
No  data  available,  not  used 

7.3  Maintenance 

7.3.1  Cost  of  Preventive  Maintenance 

7.3. 1 . 1  Annual  man  hour  costs  for  major  preventive  maintenance  actions  on  main  system 
MHCPMSYS*NHPMSYS 

MHCPMSYS  -  Man  hour  costs  for  major  preventive  maintenance  actions  on  main  system 

NHPMSYS  -  Number  of  labour  hours  for  major  preventive  maintenance  actions  on  main  system  RNLN  input 

RNLN  input 

7. 3. 1.2  Materiel  costs  for  major  preventive  maintenance  actions  on  main  system 
MCPMSYS 

RNLN  input 
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Table  8:  The  CATLOC  model  A1-2  Tree  reporting  table  (Cont’d) 

Title/Index  Description 

7.3. 1.3  Annual  man  hour  costs  for  PM  actions  at  unit  level. 

CHPMR 

NYPM*QTYPM*TPM*NMPM*MHC 

The  formula  mentioned  above  is  just  to  illustrate  how  this  cost  atom  may  be  computed. 

The  formula  does  not  work  in  the  model  as  the  parameters  should  be  defined  as  vectors  or  arrays  over  predefined  domains. 

See  section  4  for  definition  of  the  parameters.  Kept  in  the  model  for  demonstration  reasons. 

7.3. 1.4  CMPMT 
NYPM*QTYPM*CMPM 

Annual  materiel  costs  for  PM  actions  at  unit  level.  See  7. 3. 1.3. 

7.3.1 .5  Annual  costs  for  PM  actions  at  unit  level  performed  by  contractor 

CPMTC 

IF(LPM= 1  ,N YPM*QTYPM*CPTC,0)  See  7.3. 1.3. 

7.3.2  Cost  of  Corrective  Maintenance 

7.3.2. 1  Average  CM  costs  due  to  unit  replacement  actions 

7. 3. 2. 1.1  Man  hour  costs  for  unit  replacements  due  to  CM  during  operational  time  onboard  the  ship  performed 
by  NME  at  Organisational  level. 

CHCMRP 

MHC*QTYPM*FRT*ACMRP*NMTRP*MTTRP 

The  formula  mentioned  above  is  just  to  illustrate  how  this  cost  atom  may  be  computed.  The  formula  does  not  work  in  the  model 
as  the  parameters  should  be  defined  as  vectors  or  arrays  over  predefined  domains.  See  section  4  for  definition  of  the  parameters. 
Kept  in  the  model  for  demonstration  reasons. 

7. 3. 2. 1.2  Average  cost  for  materiel  consumption  (consumables)  due  to  CM  replacement  actions  at 
Organizational  level  performed  by  the  crew. 

CMCMRP 

UTILF*0.00876*QTYPM*FRT*(1-ACMC)*CMRP 

It  Is  the  aggregate  value  for  all  the  parts  which  has  a  value  for  CMRP  and  where  LRE  is  OLM  The  same  comments  as  above. 

7. 3. 2. 1.3  Average  cost  for  transportation  due  to  CM  replacement  action  at  OLM  performed  by  NME 
personnel  abroad 

CTCMRP 

QTYPM*FRT*UTILF*0.00876*ACMRP*CMTRP  The  same  comments  as  above. 

7. 3. 2. 1.4  Average  cost  for  CM  replacements  at  unit  level  performed  by  external  personnel 

CCMRPEX 

QTYPM*FRT*UTILF*0.00876*ACMC*CMRPC 

These  costs  should  include  both  man  power  costs  and  material  consumption  costs 
The  same  comments  as  above. 

7. 3. 2. 1.7  Average  transportation  costs  for  replacements  actions  due  to  CM  at  OLM  performed  by  personnel  from  NME  per  year 
-  includes  personnel  and  replaced  unit  transportation  back  and  forth 

TRCCMRP 

Should  be  an  input  from  the  user.  Kept  in  the  model  for  demonstration  reasons 

7. 3. 2. 1.8  Average  cost  for  replacement  actions  if  performed  by  contractor  /  external  man-power  at  OLM 

CMRPEX 

Should  be  an  input  from  the  user.  Kept  in  the  model  for  demonstration  reasons 

7. 3. 2. 1.5  Average  cost  of  consumables  consumption  for  replacements  due  to  CM  at  OLM  per  year 

CMRPCC 

Should  be  an  input  from  the  user.  Kept  in  the  model  for  demonstration  reasons 

7. 3. 2. 1.6  Average  man-hour  cots  for  replacements  due  to  CM  at  OLM  performed  by  personnel  from  NME 
per  year 

MHCCMRP 

Should  be  an  input  from  the  user.  Kept  in  the  model  for  demonstration  reasons 
13.2.2  Average  CM  costs  due  to  unit  repair  actions 
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Table  9:  The  CATLOC  model  A1-2  Tree  reporting  table  (Cont’d) 

Title/Index  Description 

73.2.2. 1  Average  CM  costs  for  repair  actions  at  unit  level  performed  by  NME  per  year 

CCMRNME 

0.00876*UTILF*QTYPM*FRT*(MHC*NMTR*MTTR+CMCM) 

The  above  formula  is  just  to  illustrate  how  this  cost  atom  may  be  computed. 

The  formula  does  not  work  in  the  model  as  it  is  because  the  parameters  should  be  defined 

as  vectors  or  arrays  over  predefined  domains.  Se  Ch.4  for  parameters  definitions.  Kept  in  the  model  for  demonstration  reasons 

73.2.2.2  Average  CM  costs  for  repair  actions  at  unit  level  performed  by  contractor  per  year 
CMCMCT 

QTYPM*FRT*UTILF*0.00876*CMCMC  The  same  comments  as  above. 

7.3.2.23  CM  -  Average  cost  of  consumables  for  repair  actions  due  to  CM  per  year 
CMCMCC 

Should  be  an  input  from  the  user.  Kept  in  the  model  for  demonstration  reasons 

73.2.2.4  CM  -  Average  man-hour  cots  for  repair  actions  due  to  CM  per  year 

MHCCMR 

Should  be  an  input  from  the  user.  Kept  in  the  model  for  demonstration  reasons 

73.2.2.5  CM  -  Average  cost  for  repair  actions  due  to  CM  per  year  performed  by  contractor 
CMCMEX 

Should  be  an  input  from  the  user.  Kept  in  the  model  for  demonstration  reasons 

7.3.23  Average  yearly  CM  cost,  both  material  and  man  power 
AYCMC 

Average  yearly  cost  for  CM  actions 

This  cost  is  calculated  as  a  result  of  the  assumption  made  in  Ch.  4 

7.3.3  Costs  related  with  other  operating  systems 

7.33.1  Average  man-hour  cost  for  maintenance  of  helicopters  (maintenance  staff)  NHALO*MHCOPH 
MHCOPH  -  Average  man-hour  cost  for  maintenance  of  helicopters  (maintenance  staff)  Input  data  from  RNLN 

7.33.2  Average  materiel  costs  due  to  maintenance  of  helicopter  per  year 
NHALO*CMMHY 

CMMHY  -  Average  materiel  costs  due  to  maintenance  of  helicopter  per  year 
Input  data  from  RNLN 

7.3.5  Cost  for  updating  maintenance  documentation 
NDMU*NDMY*CDMU 

NDMU  -  Number  of  maint.  documentations  updates  for  each  type  per  year  NDMY 

-  Number  of  maint.  documentations  updates  of  type  i  per  year  CDMU 

-  Cost  for  one  maint.  documentation  update  of  type  i  per  year 
No  data  available.  Not  used 

7.3.4  Cost  of  test  and  support  equipment  and  tools 
CTSEQ+CITSEQ 

CITSEQ  -  Yearly  investment  cost  for  test/support  equipment 
CTSEQ  -  Yearly  support  cost  for  test/support  equipment 
No  data  available.  Not  used 

7.3.6  Cost  of  yearly  replenishment  of  spares 

CMRPSP 

7.3.7  Costs  for  contract  logistic  support 
No  data  available.  Not  used 

7.5  PHST 

7.5.2  MHC  for  handling  of  Government  Owned  Stores 

MHCPHST 

7.5.1  Materiel  costs  for  handling  of  Government  Owned  Stores 

MCPHST 

7.4  Sustaining  Support 
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Table  10:  The  CATLOC  model  A1-2  Tree  reporting  table  (Cont’d) 

Title/Index  Description 

7.4.1  Costs  of  periodic  training  operating  personnel 
NTC_OP*NTP_OP*CTC_OP+CTOPCY 

CTC_OP  -  Cost  for  one  op.  training  course  of  type  i  per  attendee  including  the  necessary  documentation 
CTOPCY  -  Annual  cost  for  training  material  (consumables)  for  operating  personnel  [cost/year] 

NTC_OP  -  Number  of  op.  training  courses  for  each  type  per  year 

NTP_OP  -  Number  of  attendees  per  op.  training  course  of  type  i  No  data  available.  Not  used 

7.4.2  Costs  of  periodic  training  maintenance  personnel 
NTC_MAINT*NTP_MAIN*CTC_MAINT+CTMCY 

CTC_MAINT  -  Cost  for  one  maint.  training  course  of  type  i  per  attendee  including  the  necessary  documentation 
CTMCY  -  Annual  cost  for  training  material  (consumables)  for  maintenance  personnel  [cost/year] 

NTC_MAINT  -  Number  of  maint.  training  courses  for  each  type  per  year 

NTP_MAIN  -  Number  of  attendees  per  maint.  training  course  of  type  i  No  data  available.  Not  used 

7.4.3  Yearly  cost  for  training  devices 

CTD  *  SUM(NTD ,  ALL_TR  AININ  GS ) 

NTD  -  Number  of  training  devices  of  type  k  used  for  training  courses  /year 
CTD  -  Average  cost  for  using  a  device  of  type  k  once 
No  data  available.  Not  used 

7.4.4  Yearly  acquisitions  cost  for  trainings  devices 
CATD*NTDA 

CATD  -  Annual  investments  in  training  devices  of  type  k 

NTDA  -  Number  of  training  devices  acquired  per  year  No  data  available.  Not  used 

7.6  Restoration 

7.6.1  Cost  of  recommended  and  mandatory  modifications  of  the  system 

No  data  available.  Not  used 

7.7  Indirect  Support 

7.7.1  Man  hour  costs  generated  by  Navy  Command  (  Headquarters,  Personnel  &  Operations  ) 

MHCNC 

7.7.2  Materiel  costs  generated  by  Navy  Command  (  Headquarters,  Personnel  &  Operations  ) 

MCNC 

1  LCCA 

Life  Cycle  Cost  Aq. 

1.7  IEAC 

Assigned  for  independent  estimated  acquisition  cost 
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Appendix  B2-2:  EX  ANTE  TESTING  -  CAIG 


PURPOSE 

The  puipose  of  this  study  was  to  develop  a  cost  estimating  relationship  methodology  that  could  be  used  to 
estimate  the  Operating  and  Support  (O&S)  cost  for  the  Dutch  Navy  Landing  Platform  Dock  (LPD)  Rotter¬ 
dam  (L-800)  and  Johan  de  Witt  (L-801)  ships5.  The  basis  for  the  methodology  was  the  use  of  the  U.S.  Naval 
Visibility  and  Management  of  Operating  and  Support  Costs  (VAMOSC)  [5]  management  information  system. 
VAMOSC  collects  and  reports  U.S.  Navy  and  U.S.  Marine  Coips  historical  weapon  system  O&S  costs.  VA¬ 
MOSC  provides  the  direct  O&S  costs  of  weapon  systems,  some  linked  indirect  costs  (e.g.,  ship  depot  overhead), 
and  related  non-cost  information  such  as  flying  hour  metrics,  steaming  hours,  age  of  aircraft,  etc. 

DATA 

General  data  specifications  for  the  L800  and  L801  were  provided  by  Study  Director  Dr.  Brian  Flynn.  The 
information  was  verified  to  be  accurate  via  Jane’s  and  open-source  information  from  the  internet.  U.S.  ship  data 
was  pulled  from  VAMOSC,  with  all  costs  in  constant  fiscal  year  (FY)  2009  dollars.  Additional  information  was 
taken  from  the  Naval  Vessel  Registry  [6],  which  is  the  official  inventory  of  U.S.  ships  and  service  craft  from 
the  time  of  vessel  authorization  through  its  life  cycle  and  disposal.  Initially,  O&S  data  was  pulled  for  the 
following  ships:  LCC-19,  LHD-1,  LPD-1,  LPD-4,  LPD-17,  LPH-2,  LSD-36,  LSD-41,  LSD-49  and  LST-1179. 
Ultimately,  LCC-19,  LHD-1,  LPH-2  and  LST-1 179  were  dropped  from  the  data  set  because  the  dimensions  of 
these  classes  were  not  analogous  to  the  L800  and  L801.  LPD-17  was  dropped  because  there  were  only  2-years 
of  data  available  for  that  class.  Therefore,  the  ships  classes  used  in  the  final  dataset  were: 

LPD-1  LPD-4  LSD-36  LSD  41  LSD-49 

To  deal  with  the  high  variability  of  O&S  costs  between  hulls,  classes  and  years,  two  steps  were  taken.  First, 
data  was  averaged  on  an  annual  basis  between  the  different  hulls  in  each  class.  This  was  done  so  as  to  provide 
a  better  estimate  of  the  annual  O&S  costs  for  each  class.  Although  some  ships  of  a  class  may  experience 
abnormal  costs  from  year  to  year,  when  averaged  together  this  volatility  should  be  mitigated.  The  yearly  data 
for  each  class  of  ship  was  then  averaged  over  the  life  of  the  hull,  in  all  cases  beginning  in  1984  and  ending 
when  the  ship  was  either  decommissioned  or  the  end  of  the  sample  period  (2008)  was  reached.  Again,  this  was 
done  to  deal  with  the  high  variability  of  O&S  costs.  It  is  noted  that  the  presented  methodology  can  be  used  to 
estimate  the  average  annual  O&S  costs  of  a  hull,  not  the  specific  O&S  costs  in  a  given  year. 

PARAMETRIC  COST  ESTIMATING  RELATIONSHIP  METHODOLOGY 

The  parametric  method  is  based  on  various  characteristics  or  measurable  attributes  of  the  system,  hardware 
and  software  being  estimated.  It  depends  upon  the  existence  of  a  causal  relationship  between  system  costs  and 
these  attributes.  Such  relationships,  known  as  CERs  (cost  estimating  relationships)  are  typically  estimated  from 
historical  data  using  statistical  techniques. 

5  This  appendix  is  an  exact  copy  of  the  report:  Methodology  for  Approximating  Annual  Operating  Support  Cost  of  Amphibious 
Ships  -  Dutch  Navy  Landing  Platform  Dock  Rotterdam  (L800)  and  Johan  de  Witt  (L801)  by  Michael  Mender  and  John  Murray  of  the 
U.S.  Naval  Center  for  Cost  Analysis  [4], 
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The  methodology  behind  the  creation  of  the  CERs  for  each  Cost  Analysis  Improvement  Group  (CAIG)  cost 
element  was  as  follows. 

•  First,  analogous  ships  to  Rotterdam  and  Johann  de  Witt  were  selected  and  their  data  pulled  from  VA- 
MOSC. 

•  The  work  breakdown  structures  (WBS)  for  each  element  were  analysed  so  as  to  identify  likely  cost  drivers 
for  each  element. 

•  Possible  drivers  were  then  analysed  via  scatter  plots  to  evaluate  the  relationship  between  the  probable 
drivers  and  their-  respective  CAIG  elements. 

•  If  a  correlation  was  revealed,  regression  calculations  were  run  via  the  Excel  module  ‘CoStat’,  to  quantify 
the  relationship  and  allow  non-arbitrary  comparisons  of  cost  drivers. 

Although  other  functional  forms  were  evaluated,  all  of  the  final  CERs  used  standard  linear  regression  mod¬ 
els  with  one  variable. 

The  equation  therefore  takes  the  form: 


Y  =  B0  +  BlXu  (14) 

where  Y  is  the  CAIG  element  being  estimated  (the  dependent  variable),  Bq:  is  a  constant  term,  X\.  is  the 
identified  cost  driver,  and  B\ :  is  the  coefficient  attached  to  that  cost  driver. 

CAIG  1.0  (Unit  Manpower) 

This  CAIG  element  is  driven  by  the  number  of  personnel  (billets)  assigned  to  a  ship,  and  personnel  rates 
associated  with  each  billet  [7].  Billets  per  ship  were  originally  identified  as  the  main  cost  driver  to  be  used 
in  the  CER,  but  further  analysis  of  the  data  revealed  that  due  to  substantially  different  manning  requirements 
between  the  U.S.  and  Dutch  hulls,  a  regression  based  upon  the  number  of  billets  was  not  feasible.  The  problem 
being  that  all  U.S.  ships  that  are  of  a  similar  size  and  function  to  the  L-800CL  had  approximately  300  billets. 
With  all  the  data  points  clustered  around  one  number;  no  correlation  existed  to  base  a  CER.  As  a  result,  full 
load  displacement  was  selected,  however,  displacement  only  works  when  the  ships  for  which  the  estimate 
is  being  prepared  have  similar  manning  requirements  to  U.S.  ships.  As  the  L-800/L-801CL  don’t  have  similar 
manning  requirements,  the  recommendation  was  that  CAIG  1 .0  should  be  estimated  using  the  analogous  method 
presented  later  in  this  report. 

CAIG  2.0  (Unit  Operations  Cost) 

This  CAIG  element  is  largely  driven  by  fuel  costs.  Ships  that  consume  more  fuel  experience  higher  CAIG  2.0 
costs  than  ships  that  use  less  fuel.  The  decision  was  made  to  develop  a  CER  around  annual  fuel  consumption. 
As  this  piece  of  data  was  not  available  for  the  L-800/L-801CL  class  of  ships,  an  intermediate  equation  using 
full  load  displacement  and  a  dummy  variable  for  engine  type  was  used  to  estimate  annual  fuel  consumption. 
Please  note  that  this  implicitly  assumes  that  the  RNLN  has  a  similar  usage  portfolio  when  compared  to  the 
USN,  additionally  it  assumes  that  the  RNLN  experiences  comparable  engine  efficiency  and  fuel  costs. 
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CAIG  3.0  (Maintenance  Costs) 

This  CAIG  element  was  found  to  be  largely  driven  by  full  load  displacement  (FLD).  Larger  ships  cost  more  to 
maintain,  therefore,  a  simple  regression  linking  full  load  displacement  to  CAIG  3.0  was  used  to  estimate  the 
maintenance  costs  of  the  L-800  and  L-801. 

CAIG  4.0  (Sustainment  Costs) 

This  CAIG  element  was  found  to  be  similar  to  CAIG  3.0.  A  major  cost  driver  is  FLD,  and  a  correlation  was 
found  to  exist  between  an  increase  in  full  load  displacement  and  larger  sustainment  costs. 

CAIG  5.0  (Continuing  Systems  Improvements) 

This  CAIG  element  is  notoriously  difficult  to  estimate.  Upgrades  and  systems  improvements  can  vary  wildly 
between  hulls  in  a  class,  between  years  for  individual  hulls,  and  are  also  heavily  influenced  by  policy  and  other 
non-quantifiable  factors.  A  variety  of  methods  and  theories  were  evaluated,  with  the  following  being  the  only 
one  that  showed  any  promise.  It  was  hypothesized  that  ships  with  higher  annual  fuel  consumption  may  also 
have  higher  annual  CAIG  5.0  related  costs.  The  reason  we  might  expect  to  see  this  is  that  ships  that  consume 
more  fuel,  probably  have  a  higher  OPTEMPO  and  may  receive  higher  priority  for  upgrades  as  a  result.  Keeping 
in  mind  that  upgrades  are  influenced  by  policy  and  other  factors,  it  is  likely  that  the  actual  CAIG  5.0  costs  for 
the  RNLN  (or  any  navy  other  than  the  USN)  will  differ  significantly. 

MONTE  CARLO  SIMULATION  LOR  PREDICTION  INTERVALS 

Once  the  CERs  for  each  individual  CAIG  element  were  established,  the  combined  information  was  assembled 
into  an  estimate  through  the  use  of  CoStat  and  Crystal  Ball.  CoStat  can  be  programmed  to  automatically 
generate  prediction  intervals  when  a  regression  is  run.  By  setting  the  prediction  intervals  such  that  they  were  one 
standard  deviation  from  the  mean,  the  standard  deviation  of  each  estimate  was  easily  computed  via  simple  math. 
This  information  was  then  fed  into  Crystal  Ball.  Crystal  Ball  allows  analysts  to  run  Monte  Carlo  simulations, 
which  allow  for  the  generation  of  prediction  intervals  for  the  combined  estimate.  This  is  done  by  using  the 
mean  estimate  and  standard  deviation  for  each  CAIG  element  to  create  a  normal  distribution  for  that  element. 
The  combined  estimate  was  then  set  equal  to  the  sum  of  the  most  probable  values  of  each  of  the  elements. 
Crystal  Ball  then  runs  a  simulation,  during  which  random  numbers  are  generated  according  to  the  probabilities 
associated  with  the  normal  distributions  for  each  element.  After  200,000  iterations,  a  histogram  of  the  results 
reveals  the  most  likely  values  of  the  combined  estimate. 

ANALOGOUS  COST  ESTIMATING  METHODOLOGY 

First,  of  all  the  classes  in  the  available  dataset,  the  most  analogous  based  on  its  similarities  in  dimensions  to  the 
Rotterdam  and  Johan  de  Witt  was  selected.  Then  the  analogous  ratio  or  multiplier  defined  for  each  cost  element 
e  will  be: 

Re  —  Ae/ Pg  ,  (15) 

where  Ae  is  the  historical  average  analogous  system  cost  corresponding  to  the  CAIG  cost  element  e  being 
estimated  (the  dependent  variable),  and  P}  is  the  value  of  the  analogous  ship  parameter  on  which  the  analogy 
will  be  made  for  the  CAIG  cost  element  e  being  estimated. 
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The  estimated  cost  for  the  cost  element  e  is: 


Ye  =  RexP°,  (16) 

where  P{J  is  the  value  of  the  estimated  ship  parameter  on  which  the  analogy  will  be  made  for  the  CAIG  cost 
element  e  being  estimated. 

Of  the  five  classes  in  the  dataset,  the  LSD-49  Class  was  the  most  analogous  based  on  its  similarities  in 
dimensions  to  the  L-801  but  is  slightly  larger  than  L-800. 


Table  11:  Ship  parameters 


Ship  Name 

Rotterdam 

Johan  de  Witt 

Harpers  Ferry 

Nation 

Netherlands 

Netherlands 

United  States 

Number 

L800 

L801 

LSD  49 

Type 

LPD 

LPD 

Landing  Ship  Dock 

Dimensions 

Length(m) 

162.2 

176.35 

185.9 

Beam(m) 

25 

25 

25.6 

Draught(m) 

5.9 

5.9 

6.3 

Displacement(tonnes) 

Light 

? 

? 

11604 

Full  Load 

12750 

16680 

16601 

Crew  Complement 

Officers 

13 

17 

30 

Crew 

100 

129 

309 

Additionally,  the  LSD-49CL  is  the  newest  U.S.  amphibious  ships  among  the  VAMOSC  dataset,  and  it  has 
a  similar  power  train  (Diesel  Electric)  to  the  L-800  and  L-801.  The  class  consists  of  the  following  hulls: 

LSD  -  49  LSD -50  LSD  -51  LSD  -52. 

With  the  exception  of  CAIG  1.0  elements,  the  methodology  for  developing  the  analogous  cost  estimate  was 
based  on  each  Dutch  LPD’s  weight  relative  to  the  LSD-49CL.  For  CAIG  elements  2.0  -  5.0,  the  equation  takes 
the  form: 


Y  =A0x(FLD0/FLD{)  ,  (17) 

where  Y  is  the  CAIG  element  being  estimated  (the  dependent  variable),  Aq  is  the  historical  average  LSD-49 
class  cost  in  CY09$  corresponding  to  the  CAIG  element  being  estimated,  FLDq  is  the  L-800  or  L-801  full  load 
displacement  (tonnes),  and,  FLD\  is  the  LSD-49CL  average  full  load  displacement  (tonnes). 

2-5.1  CAIG  1.0  (Unit  Manpower) 

The  L-800  and  L-801  do  not  have  similar  manning  requirements.  The  recommendation  for  estimating  CAIG  1.0 
is  to  multiply  the  estimated  number  of  billets  with  the  expected  personnel  rates.  Personnel  rates  were  obtained 
by  averaging  the  annual  cost  per  billet  for  officers  (0-5  to  WO-2)  and  for  enlisted  (E-9  to  E-l)  into  a  composite 


RTO-TR-SAS-076 


B2  -49 


ANNEX  B2  -  ESTIMATING  THE  OPERATION  AND 
SUPPORT  COSTS  OF  THE  ROYAL  NETHERLANDS  NAVY 
LANDING  PLATFORM  DOCK  SHIPS:  AN  EX  ANTE  ANALYSIS 


ORGANIZATION 


|  CY09S  | 


O&S  CAIG  Cost  Elements 

O&S  CAIG 

1.0  Unit  Level  Manpower 

Cost  Elements 

2.0  Unit  Operations 

have  been 

3.0  Maintenance 

sanitized  due 

4.0  Sustaining  Support 

to  business 

5.0  Continuing  System  Improvements 

sensitivity 

Total  O&S  Cost  (CY$09M) 

Rotterdam 

CAIG  1.0 

9,591,412 

CAIG  20 

4,020,534 

CAIG  3.0 

7,173,284 

CAIG  4.0 

949,369 

CAIG  5.0 

1,705,253 

23,439,852 

Johan  de  Witt 

CAIG  1.0 

12404,367 

CAIG  20 

5,259,804 

CAIG  3.0 

9,384,343 

CAIG  4.0 

1,241,998 

CAIG  5.0 

2230,873 

30,521,385 


Figure  16:  VAMOSC  O&S  cost  estimate  results  for  HMS  Rotterdam  and  Johan  de  Witt 


officer  rate  and  an  enlisted  composite  rate.  These  two  numbers  were  then  multiplied  by  the  number  of  officer 
billets  and  enlisted  billets  per  ship,  as  a  means  of  estimating  the  annual  unit  manpower  costs  per  ship. 

Due  to  the  significantly  smaller  number  of  billets  of  the  RNLN  ships,  there  will  be  a  significant  difference 
in  CAIG  1.0  between  USN  and  RNLN  ships.  The  method  presented  in  this  section  was  designed  to  capture  that 
difference  as  it  includes  the  number  of  billets  per  ship,  and  the  only  assumption  is  that  the  RNLN  experiences 
similar  personnel  costs. 
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Appendix  B2-3:  COST  BREAKDOWN  STRUCTURE 
FOR  THE  GCBS-CAIG  MODEL 

-  O  I:\FMV\SAFEHO\Uppgifter_Logmodeller\l  Metodstod\SA5  076  medverkan\Arbetslage\GCB5_ver5.cci 

e  El  GCBS _ 


jO&S  GCBS-CAIG,  Operational  and  Support  Costs 


S  ■  1,  Operation 

®  ■  7.1.1,  Personnel  costs 
a  ■  7.1.2,  Petroleum.  Oil  and  Lubricants 
4  0  7. 1.2.1,  Cost  of  fuel 
l~l  7. 1.2.2,  Cost  of  oi  consumption 
Q  7. 1.2.3,  Cost  of  lubricants 
4  □  7. 1.2.4,  Yearly  costs  for  POL,  Ship 

4  □  7. 1.2.5,  Average  POL  costs  due  to  maintenance  of  helicopter  per  year 
B  ■  7.1.3,  Munitions'  Costs 

4  □  7. 1.3.1,  Costs  of  munitions  and  countermeasures  consumption 
a  ■  7.1.4,  Cost  of  other  consumables  (food,  water,  soft  drinks,  etc) 

4  □  7.1. 4.1,  Average  cost  for  food  per  year 
4  O  7. 1.4.2,  Average  yearly  cost  for  water  consumption 
4  □  7. 1.4.3,  Average  yearly  cost  for  electricity  consumption 
4  0  7. 1.4.4,  Average  yearly  cost  of  "other  operational  material” 

H  ■  2,  Supporting  operations 

4  0  7.2.1,  Costs  of  updating  operating  documentation. 

4  0  7.2.2,  Costs  of  use  infrastructure  for  operating  the  system 
0  7.2.3,  Costs  updating  computer  software/hardware  for  information  systems. 

0  7.2.4,  Cost  of  embedded  softwares 
h|3,  Maintenance 

s  ■  7.3.1,  Cost  of  Preventive  Maintenance 

4  0  7.3. 1 . 1,  Annual  man  hour  costs  for  major  preventive  maintenance  action  on  main  system 
4  0  7.3. 1 .2,  Materiel  costs  for  major  preventive  maintenance  action  on  main  system 
s  ■  7.3.2,  Cost  of  Corrective  Maintenance 

O  7.3.2. 1,  Average  CM  costs  due  to  unit  replacement  actions 
0  7.3.2.2,  Average  CM  costs  due  to  unit  repair  actions 
4  0  7.3.2. 3,  Average  yearly  CM  cost ,  both  material  and  man  power 
B  ■  7.3.3,  Costs  related  with  other  operating  systems 

4  0  7.3.3. 1,  Average  man-hour  cost  for  maintenance  of  helicopters  (maintenance  staff) 

4  0  7.3.3. 2,  Average  materiel  costs  due  to  maintenance  of  helicopter  per  year 
4  0  7.3.6,  Cost  of  yearly  replenishment  of  spares 
0  7.3.7,  Costs  for  contract  logistic  support 
B  ■  4,  Sustatining  Support 

*  0  7.4.1,  Costs  of  periodic  training  operating  personnel 

4  0  7.4.2,  Costs  of  periodic  training  maintenance  personnel 

*  0  7.4.3,  Cost  of  test  and  support  equipment  and  tools 

*  0  7.4.4,  Cost  for  updating  maintenance  documentation 

4  0  7.4.5,  Materiel  costs  for  handling  of  Government  Owned  Stores 
®  0  7.4.6,  MHC  for  handling  of  Government  Owned  Stores 
B  |  5,  Restoration 

O  7.5.1,  Cost  of  recommended  and  mandatory  modifications  of  the  system 
B  |  6,  Indirect  Support 

*  0  7.6.1,  Man  hour  costs  generated  by  Navy  Command  (  Headquarters,  Personnel  &  Operations  ) 

*  0  7.6.2,  Materiel  costs  generated  by  Navy  Command  (  Headquarters,  Personnel  &  Operations  ) 

*  0  7.6.3,  General  Training  and  Education 
#  El  Default  View 

Figure  17:  Cost  Breakdown  Structure  for  the  GCBS-CAIG  model 
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ORGANIZATION 


Appendix  B2-4:  COST  BREAKDOWN  STRUCTURE 
FOR  THE  CAIG  MODEL 


h  □  I:\FMV\SAFEHD\Uppgifter_Logmodeller\l  MetodstOd\SAS  076  medverkan\Systecon\Leverans\CAIG  model  -  no  values.cd 
B  O  CES 

a  ■  OSD  CAIG  O&S,  Systems  Operating  and  support  cost 

a  ■  1.0  UNIT-LEVEL  MANPOWER,  Unit-level  manpower  costs 

□  1.1  OPERATIONS  MANPOWER,  Cost  of  operations  manpower 

□  1.2  UNIT-LEVEL  MAINTENANCE  MANPOWER,  Cost  of  unit-level  maintenance  manpower 

□  1.3  OTHER  UNIT-LEVEL  MANPOWER,  Cost  of  other  unit-level  manpower 
a  ■  2.0  UNIT  OPERATIONS,  Unit  operations  costs 

(r  ■  2. 1  OPERATING  MATERIAL,  Cost  of  operating  material 

□  2.1.1  ENERGY,  Cost  of  energy  (POL) 

I  I  2.1.2  TRAINING  MUNITIONS  &  EXP  STORES,  Cost  of  training  munitions  and  expendable  stores 
I  I  2.1.3  OTHER  OPERATIONAL  MATERIAL,  Cost  of  other  operational  material 

□  2.2  SUPPORT  SERVICES,  Cost  of  support  services 

□  2.3  TEMPORARY  DUTY,  Cost  of  temporary  duty 
a  ■  3.0  MAINTENANCE,  Maintenance  costs 

a  ■  3.1  ORG  MAINTENANCE  AND  SUPPORT,  Cost  of  org  maintenance 

□  3.1.1  ORG  LEVEL  CONSUMABLES,  Cost  of  org  level  consumables 

□  3.1.2  ORG  LEVEL  REPAIR  PARTS,  Cost  of  org  level  repair  parts  (DU) 

I  I  3.1.3  ORG  LEVEL  DLR:S,  Cost  of  org  depot  level  reparables  (LRU) 

n  3.1.4  CONTRACT  MAINTENANCE  SERVICES,  Cost  of  contract  maintenance  services 

□  3.1.5  OTHER  UNIT  MAINTENANCE,  Cost  of  other  unit  maintenance 

-  ■  3.2  INTERMEDIATE  MAINT  AND  SUPPORT,  Cost  of  intermediate  maint  and  support 
I  I  3.2.1  INTERM  LEVEL  CONSUMABLES,  Cost  of  intermediate  level  consumables 

□  3.2.2  INTERM  LEVEL  REPAIR  PARTS,  Cost  of  intermediate  level  repair  parts  (DU) 
n  3.2.3  INTERM  LEVEL  DLR:S,  Cost  of  intermediate  depot  level  reparables  (LRU) 

I  I  3.2.4  GOVERNMENT  LABOR,  Cost  of  manpower  performing  intermediate  maintenance 
I  I  3.2.5  CONTRACTOR  MAINTENANCE,  Cost  of  contractor  maint  at  intermediate  level 
HI  3.2.6  OTHER  INTERMEDIATE  MAINTENANCE,  Cost  of  other  intermediate  maintenance 
S  ■  4.0  SUSTAINING  SUPPORT,  Sustaining  support  costs 

—  H  4. 1  SYSTEM  SPECIFIC  TRAINING,  Cost  of  system  specific  training 

□  4.1.1  SYSTEM  SPECIFIC  OPERATOR  TRAINING,  Cost  of  system  specific  operator  training 

□  4.1.2  SYSTEM  SPECIFIC  NON-OPERATOR  TRAINING,  Cost  of  system  specific  non-operator  training 

□  4.2  SUPPORT  EQUIPMENT  REPLACEMENT,  Cost  of  support  equipment  replacement 

□  4.3  OPERATING  EQUIPMENT  REPLACEMENT,  Cost  of  operating  equipment  replacement 

I  I  4.4  SUSTAINING  ENGINEERING  ANO  PROGRAM  MGMT,  Cost  of  sustaining  engineering  and  program  mgmt 
n  4.5  OTHER  SUSTAINING  SUPPORT,  Cost  of  other  sustaining  support 
S  ■  5.0  CONTINUING  SYSTEM  IMPROVEMENTS,  Continuing  support  costs 

F-  ■  5.2  SOFTWARE  MAINTENANCE  AND  MODIFICATIONS,  Cost  of  sw  maintenance  and  modifications 

□  5.2.1  CORRECTIONS  OF  DEFICIENCIES,  Cost  of  deficiencies 

□  5.2.2  SOFTWARE  ENHANCEMENTS,  Cost  of  software  enhancements _ 


5.1  HARDWARE  MODIFICATIONS  OR  MODERNIZATIONS,  Cost  of  hw  modifications  or  mondernization 


i-  ■  6.0  INDIRECT  SUPPORT,  Indirect  support  costs 

B  ■  6.2  PERSONNEL  SUPPORT,  Cost  of  personnel  support 

H  ■  6.2. 1  PERSONNEL  ADMINISTRATION,  Cost  of  personnel  administration 

□  6.2. 1.1  PERSONNEL  ACQUISITION,  Cost  of  personnel  acquisition 

□  6.2. 1 .2  PERSONNEL  NOT  AVAILABLE  FOR  DUTY,  Cost  of  personnel  not  available  for  duty 
F-  ■  6.2.2  PERSONNEL  BENEFITS,  Cost  of  personnel  benefits 

□  6.2.2. 1  FAMILY  HOUSING,  Cost  of  familiy  housing 

□  6. 2. 2. 2  DEPENDENT  SUPPORT,  Cost  of  dependent  support 

n  6. 2.2. 3  COMMISSARIES  AND  EXCHANGES,  Cost  of  commissaries  and  exchanges 
□  6.2.3  MEDICAL  SUPPORT,  Cost  of  medical  support 

□  6.1  INSTALLATION  SUPPORT,  Cost  of  Installation  support 

□  6.3  GENERAL  TRAINING  ANO  EDUCATION,  Cost  of  general  training  and  education 
B  □  Default  View 

*  ■  All  cost  atoms 


Figure  18:  Cost  breakdown  structure  for  the  CAIG  model 
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Second-Level  Cost  Estimating  Structure 


1.  Unit-Level  Manpower 

1.1  Operations 

1.2  Unit-Level  Maintenance 
1.  3  Other  Unit-Level 

2.  Unit  Operations 

2.1  Operating  Material 

2.2  Support  Services 

2.3  Temporary  Duty 

3.  Maintenance 

3.1  Organizational  Maintenance 

3.2  Intermediate  Maintenance 

3.3  Depot  Maintenance 

4.  Sustaining  Support 

4.1  System  Specific  Training 

4.2  Support  Equipment  Replacement 

4.3  Sustaining  Engineering  and  Program 

Management 

4.4  Other  Sustaining  Suppoit 


5.  Continuing  System  Improvements 

5.1  Hardware  Modifications  or  Modernization 

5.2  Software  Maintenance  and  Modifications 

6.  Indirect  Support 

6.1  Installation  Support 

6.2  Personnel  Support 

6.2.1  Personnel  Administration 
(Personnel  Acquisition , 

Individuals  Overhead  Accounts) 

6.2.2  Personnel  Benefits 
(Family  Housing,  Commissaries, 

Child  &  Family  Support,  DoD  Schools) 

6.2.3  Medical  Support 

6.3  General  Training  and  Education 

6.3.1  Basic  &  Initial  Skill  Training 

6.3.2  Educational  Activities 


Figure  19:  Second-level  cost  estimating  structure 

Source:  Office  of  the  Secretary  of  Defense  Cost  Analysis  Improvement  Group  (CAIG)  [8]. 
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Appendix  B2-5:  TREE  REPORT  TABLE  FOR  GCBS-CAIG  MODEL 


Table  12:  Tree  Report  Table  for  GCBS-CAIG  Model  (Cont’d  on  next  page) 

Title/Index 

Description 

1 

7.1.1 

7.1. 1.1 

7.1. 1.2 

7.1.2 

7. 1.2.1 

7. 1.2.2 

7. 1.2.3 

7. 1.2.4 

7. 1.2. 5 

7.1.3 

7. 1.3.1 

7.1.4 

7. 1.4.1 

7. 1.4.2 

7. 1.4.3 

7. 1.4.4 

2 

7.2.1 

7.2.2 

7.2.3 

7.2.4 

3 

7.3.1 

7.3. 1.1 

7.3. 1.2 

7.3.2 

73.23 

7.3.2.1 

13.2.2 

7.3.3 

7.3.3.1 

133.2 

7.3.6 

7.3.7 

Operation 

Personnel  costs 

Costs  of  operating  personnel 

Average  personnel  costs  for  operation  of  helicopters 

Petroleum.  Oil  and  Lubricants 

Cost  of  fuel 

Cost  of  oil  consumption 

Cost  of  lubricants 

Yearly  costs  for  POL,  Ship 

Average  POL  costs  due  to  maintenance  of  helicopter  per  year 

Munitions’  Costs 

Costs  of  munitions  and  countermeasures  consumption 

Cost  of  other  consumables  (food,  water,  soft  drinks,  etc) 

Average  cost  for  food  per  year 

Average  yearly  cost  for  water  consumption 

Average  yearly  cost  for  electricity  consumption 

Average  yearly  cost  of  "other  operational  material" 

Supporting  operations 

Costs  of  updating  operating  documentation. 

Costs  of  use  infrastructure  for  operating  the  system 

Costs  updating  computer  software/hardware  for  information  systems. 

Cost  of  embedded  software 

Maintenance 

Cost  of  Preventive  Maintenance 

Annual  man  hour  costs  for  major  preventive  maintenance  action  on  main  system 

Materiel  costs  for  major  preventive  maintenance  action  on  main  system 

Cost  of  Corrective  Maintenance 

Average  yearly  CM  cost ,  both  material  and  man  power 

Average  CM  costs  due  to  unit  replacement  actions 

Average  CM  costs  due  to  unit  repair  actions 

Costs  related  with  other  operating  systems 

Average  man-hour  cost  for  maintenance  of  helicopters  (maintenance  staff) 

Average  materiel  costs  due  to  maintenance  of  helicopter  per  year 

Cost  of  yearly  replenishment  of  spares 

Costs  for  contract  logistic  support 

B2-54 


RTO-TR-SAS-076 


ANNEX  B2  -  ESTIMATING  THE  OPERATION  AND 
SUPPORT  COSTS  OF  THE  ROYAL  NETHERLANDS  NAVY 
LANDING  PLATFORM  DOCK  SHIPS:  AN  EX  ANTE  ANALYSIS 


Table  13:  Tree  Report  Table  for  GCBS-CAIG  Model  (Cont’d) 

Title/Index  Description 


4  Sustaining  Support 

7.4. 1  Costs  of  periodic  training  operating  personnel 

7.4.2  Costs  of  periodic  training  maintenance  personnel 

7.4.3  Cost  of  test  and  support  equipment  and  tools 

7.4.5  Materiel  costs  for  handling  of  Government  Owned  Stores 

7.4.6  MHC  for  handling  of  Government  Owned  Stores 

7.4.4  Cost  for  updating  maintenance  documentation 

5  Restoration 

7.5. 1  Cost  of  recommended  and  mandatory  modifications  of  the  system 

6  Indirect  Support 

7.6. 1  Man  hour  costs  generated  by  Navy  Command  (  Headquarters,  Personnel  &  Operations  ) 

7.6.2  Materiel  costs  generated  by  Navy  Command  (  Headquarters,  Personnel  &  Operations  ) 

7.6.3  General  Training  and  Education 
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ANNEX  C  -  THE  ROLE  OF  LIFE  CYCLE  COST 
ANALYSIS  IN  MANAGING  THE  DEFENSE  ENTERPRISE 


1.0  INTRODUCTION 

1.1  Background 

Confronted  with  expanding  and  evolving  threats  to  our  national  security,  which  range  from  violent  extremist 
groups  equipped  with  roadside  bombs  to  hostile  nations  with  large  standing  armies  of  conscripted  soldiers  and 
weapons  of  mass  destruction,  the  fundamental  task  of  the  North  Atlantic  Treaty  Organization  (NATO),  members 
states  therein,  and  Partnership  for  Peace  (PfP)  nations,  in  its  bare  essence,  is  to  manage  risk.  With  limited 
defense  budgets  throughout  the  Alliance,  it’s  impossible  for  NATO,  its  members,  and  its  allies  to  build,  equip, 
maintain  and  deploy  the  forces  required  to  meet  all  of  its  joint  security  challenges  with  perfect  and  complete 
confidence.  The  challenges,  such  as  fighting  the  long  war  on  terror,  protecting  our  homelands  and  that  of  our 
allies,  and  preparing  to  fight  and  win  a  large-scale  conventional  campaign  against  a  near-peer  adversary,  are 
simply  too  onerous.  Instead,  hard  choices  must  be  made  amongst  competing  needs,  with  risks  driven  to  an 
acceptable  level. 

In  a  national  security  environment,  the  business  of  risk  management  involves  dimensions  of  difficulty  which 
are  often  understood  with  little  precision.  As  Admiral  Mullen,  Chairman  of  the  United  States’  Joint  Chiefs 
of  Staff,  observes  “. .  .we  live  on  the  cusp  of  a  new  era  . .  .plagued  by  uncertainty,  change,  and  unrestricted 
warfare.”[l].  To  complicate  matters,  an  entire  set  of  threats  to  security  must  be  identified  and  addressed  not  only 
today  but  several  decades  in  advance  given  the  time  needed  to  nurture  and  harvest  promising  technologies  that 
could  yield  a  comparative  military  advantage  and  to  develop,  produce,  and  field  major  new  weapons  systems 
with  required  capabilities.  The  Joint  Strike  Fighter  (JSF)  aircraft,  for  example,  took  roughly  ten  years  to  design 
and  develop,  and  will  be  in  production  through  roughly  2035,  with  a  service  life  of  decades  for  each  aircraft. 
By  simple  arithmetic,  then,  the  JSF  will  offer  a  range  of  capabilities  for  deterring  and  defeating  threats,  but  of 
uncertain  and  mutable  nature,  for  much  of  the  21st  century. 

Threats,  of  course,  can  never  be  anticipated  or  understood  perfectly.  Hopefully,  they  can  at  least  be  bounded. 
John  Maynard  Keynes,  in  discussing  investment  decision  of  entrepreneurs,  noted  “. .  .the  extreme  precariousness 
of  the  basis  of  knowledge  on  which  our  estimates  of  prospective  yield  have  to  be  made.” [2].  This  observation 
on  uncertainty  often  applies  as  well  to  assessments  of  threats,  requirements,  and  capabilities  in  the  national 
security  domain.  In  defending  our  nations,  however,  the  loss  function  for  a  wrong  decision  can  be  infinitely 
higher  than  for  the  private  sector.  Egregiously  bad  judgment  on  Bay  Street1  can  lose  a  company  but  in  national 
security  it  can  end  civilization. 

To  pile  on  difficulty,  the  defense  environment  is  highly  dynamic.  Our  enemies  can  and  do  react  to  any 
changes  in  our  capabilities,  and  are  quick  to  exploit  weaknesses.  This  holds  at  the  tactical  level,  with  the 
evolution  of  improvised  explosive  devices  in  Iraq  and  Afghanistan  but  one  example,  as  well  as  at  the  strategic, 
with  potential  adversaries  now  seeking  to  develop  anti-satellite  capabilities. 

1.2  Study  Objectives 

Given  this  extraordinarily  complex,  high-stakes  backdrop  of  strategic  planning,  capability  development,  and 
risk  mitigation,  the  SAS-076  Task  Group,  under  the  authority  of  NATO’s  Research  and  Technology  Orga¬ 
nization,  embarked  upon  an  effort  to  capture  national  practices  in  managing  the  defense  enterprise,  with  an 
emphasis  on  the  role  played  by  life-cycle  cost  analysis  in  the  process.  These  templates  were  developed  to 
ensure  a  common  denominator  of  national  responses: 

1  Bay  Street,  Toronto;  the  “Financial  Capital”  of  Canada 
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•  Overview 

Provides  an  overview  of  the  basic  systems  or  processes  used  in  a  nation  in  managing  the  defense  enterprise; 

•  Strategic  Framework 

Captures  national  practices  in  defining,  managing,  and  implementing  national  security  strategy; 

•  Needs  and  Solutions 

Defines  the  process  for  identifying  military  needs  and  developing  and  acquiring  solutions; 

•  Lexicon  and  Taxonomy 

Captures  national  definitions  of  “defense  capability”  and  defines  methods  that  might  be  employed  in  group¬ 
ing  capabilities  for  the  puipose  of  analysis;  and 

•  Role  of  Life-Cycle  Cost  Analysis 

Captures  the  degree  to  which  life-cycle  cost  analysis  plays  a  role  in  planning,  acquisition,  and  budgeting. 

As  a  corollary,  SAS-076  also  examined  a  pilot  effort  in  capability  portfolio  analysis  (CPA)  in  the  United 
States  which  investigated  how  costs,  capabilities,  and  risks  could  be  examined  together  in  an  attempt  to  engen¬ 
der  more  informed  resource  allocation  decision  making. 

In  both  cases,  for  the  templates  and  CPA,  best  practices  were  indentified  by  the  SAS-076  Task  Group. 
Recommendations  are  presented  herein. 

1.3  Scope 

SAS-076’s  expertise  lies  in  the  discipline  of  defense  cost  analysis.  That  said,  many  members  of  the  group 
have  considerable  expertise  in  defense  planning.  Therefore,  within  the  overall  context  of  managing  the  defense 
enterprise,  the  scope  of  this  effort  was  limited  by  a  desire  to  focus  on  cost-analysis  issues. 

Although  responses  from  a  larger  sample  of  nations  would  have  been  desirable,  the  responses  obtained  were 
thought  to  be  sufficient  for  generating  good  results,  lessons  learned,  and  best  practices. 

In  summary,  then,  SAS-076  offers  no  recommendations  on  critical  issues  such  as  strategic  planning,  capability- 
based  analyses,  and  on  how  best  to  acquire  materiel  solutions.  Instead,  the  role  played  by  cost  analysis  in  these 
activities  is  noted. 
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2.0  MANAGEMENT  OF  THE  DEFENSE  ENTERPRISE 


Figure  1  is  a  useful  graphic  for  describing  the  management  of  the  defense  enterprise  [3] . 


Strategic  Defense  Management  Loop 


Figure  1 :  The  strategic  defense  management  loop 


In  NATO  and  PfP  nations,  political  leaders  at  the  highest  level  proffer  national  security  guidance  or  objec¬ 
tives,  usually  in  somewhat  lofty,  broad  terms.  Departments  and  ministries  of  defense  translate  this  guidance  into 
more  concrete  policy  parameters  for  building  and  developing  their  military  forces.  In  this  context,  requirements 
for  many  countries  are  defined  in  term  of  defense  capabilities.  This  is  the  “gold  standard”  in  NATO  today.  Many 
countries  measure  performance  of  acquisition  programs,  budgets,  and  military  equipment  and  forces.  Fewer, 
however,  seem  to  measure  performance  at  a  strategic  level,  although  the  trend  is  in  this  direction. 
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3.0  CAPABILITY  PORTFOLIO  ANALYSIS 
3.1  Background 

Many  countries  within  NATO  are  interested  in  exploring  the  methods,  processes,  and  potential  benefits  of 
capability  portfolio  analysis  to  engender  better  decisions  on  the  allocation  of  scarce  defense  resources.  Our 
first  step  in  SAS-076  was  to  develop  a  deeper  understanding  of  the  scope  and  substance  of  portfolio  analysis. 
As  a  working  hypothesis,  we  defined  this  discipline,  in  a  national  security  context,  as  the  art  and  science  of 
allocating  scarce  resources  to  satisfy  strategic  requirements. 

A  literature  review  revealed  that  portfolio  analysis  in  the  private  sector  is  characterized  by 

•  Uncertain  and  changing  information, 

•  Multiple  goals  and  strategic  considerations, 

•  Interdependence  among  projects,  and 

•  Multiple  decision-makers  and  locations  [4]. 

Many  of  these  attributes  resonate  strongly  in  the  national  security  arena.  Information,  especially  in  wartime, 
is  often  uncertain  and,  at  times,  tragically  unreliable2.  And  the  business  of  defense  is  certainly  dynamic.  Threats 
change  and  induce  changes  in  requirements  and  the  value  of  existing  defense  assets.  Further,  most  of  the  ships, 
aircraft,  tanks,  and  helicopters  in  a  nation’s  arsenal  have  multiple  missions  and  capabilities,  and  are  highly 
interdependent  with  other  systems.  And  there’s  certainly  no  shortage  of  decision  makers  in  a  ministry  or 
department  of  defense,  or  within  NATO  HQ. 

We  also  learned  that  major  goals  of  portfolio  analysis  typically  include 

•  Maximizing  value  of  individual  projects, 

•  Balancing  investments,  and 

•  Adhering  to  strategy  [4]. 

In  the  private  sector,  return  on  investment  for  a  project  is  often  computed  as  net  present  value  of  future 
earnings  divided  by  investment  dollars.  In  a  defense  environment,  however,  the  appropriate  computation  might 
be  future  flows  of  military  capability  divided  by  life-cycle  costs,  with  the  numerator  a  much  more  amoiphous 
concept  than  units  of  money.  British  defense  analysts  define  military  capability  as 

“. .  .the  enduring  ability  to  generate  a  desired  operational  outcome  or  effect,  and  is  relative 
to  the  threat,  physical  environment,  and  contributions  of  coalition  partners.”  [5] 

Balance  of  a  portfolio  in  the  private  sector  is  often  assessed  along  the  dimensions  of  research  versus  pro¬ 
duction,  risk,  product  categories,  and  time  horizon.  These  vectors  hold  in  national  defense,  too.  Each  ministry 
of  defense  needs  to  balance  funding  for  basic  research,  acquisition  of  new  equipment,  and  support  and  opera¬ 
tion  of  fielded  forces.  Short-run  requirements  such  as  better  mine-resistant,  ambush-protected  vehicles  must  be 
balanced  against  investments  in  long-term  capability,  such  as  Joint  Strike  Fighter. 

2See  Ambush  Alley:  The  Most  Extraordinary  Battle  of  the  Iraq  War,  by  British  journalist  Tim  Pritchard,  for  a  heart-rending  example. 
On  23  March  2003  in  Iraq,  Clausewitz’s  "fog  of  war"  quickly  engulfed  the  1st  Battalion,  2nd  U.S.  Marines.  Due  to  an  unpredictable 
sequence  of  events  on  the  battlefield,  which  rendered  operational  plans  obsolete,  Charlie  Company  was  fired  upon  by  U.S.  Air  Force 
A-10  Warthogs  who  had  been  mistakenly  told  there  were  no  friendly  forces  around  the  northern  bridge  in  Nasiriyah. 
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Finally,  the  Chief  Executive  Officers  (CEOs)  of  best-practitioning  firms  want  to  ensure  that  projects  are 
selected  that  adhere  to  the  long-term  goals  of  the  company.  Indeed,  the  estimation  of  a  project’s  value  sometimes 
includes  an  assessment  of  its  strategic  importance  [4],  This  “on-strategy”  objective  is  equally  fitting  for  defense. 

3.2  International  CPA  Conference 

After  the  literature  review,  our  task  group  conducted  a  Capability  Portfolio  Analysis  conference  in  Paris,  France, 
in  an  effort  to  learn  more  about  the  application  of  the  discipline  in  the  international  defense  establishment.  We 
found  that  many  existing  models  and  processes  within  NATO  fall  short  of  the  ideal  goal  of  addressing  all 
strategic  requirements  and  the  capabilities  and  costs  of  all  components  of  the  portfolio,  large  and  small  alike. 

One  exception  was  the  Strategy-to-Tasks  Model  (STAM)  of  United  States  Special  Operations  Command 
(USSOCOM),  headquartered  at  MacDill  Air  Force  Base  in  Florida.  USSOCOM  is  the  combatant  command  for 
the  worldwide  use  of  Special  Forces  such  as  Navy  Seals  and  Delta  Operators3.  STAM  identifies  and  weighs 
the  importance  of  strategic  requirements  and  operational  and  tactical  tasks  required  to  meet  mission  objectives. 
A  board  of  directors  assesses  the  military  value  of  all  assets  in  the  context  of  these  requirements  and  tasks. 
STAM  does  a  good  job  of  matching  budget  to  strategy.  And  it  uses  impressive  methodology  in  evaluating 
the  warfighting  value  of  assets  in  the  USSOCOM  portfolio.  However,  USSOCOM,  unlike  most  ministries  of 
defense  within  the  Alliance,  is  not  in  the  business  of  acquiring  multi-billion  dollar  weapons  systems  since  these 
are  resourced  through  the  Services  in  the  U.S.  DoD. 


3  See  Inside  Delta  Force,  Delacorte  Press,  2002,  Erik  Haney,  for  a  first-hand  description  of  the  skills  and  valor  of  1st  Special  Forces 
Operational  Detachment  -  Delta. 
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4.0  U.S.  PILOT  EFFORT  IN  PORTFOLIO  ANALYSIS 

4.1  Background 

Applying  theory  to  practice,  the  U.S.  Department  of  the  Navy  (DON)  several  years  ago  launched  a  pilot  program 
for  building  a  construct  to  actually  do  capability  portfolio  analysis  [6].  After  considerable  debate,  the  CPA  team 
selected  the  defensive  part  of  mine  warfare,  or  mine  countermeasures,  as  the  subject  of  analysis.  The  offensive 
side,  the  deployment  of  mines,  is  of  relatively  low  dollar  value.  The  defensive  side,  on  the  other  hand,  was 
regarded  as  small  enough  to  increase  the  probability  of  success  of  the  effort  yet  large  enough  to  maximize 
lessons  learned. 

To  keep  the  pilot  program  within  manageable  scope,  the  team  decided  early  on  to  restrict  most  of  the  work 
to  the  “materiel”  component  of  the  U.S.’s  “DOTMLPF”  solution  space  of  “doctrine,  organization,  training, 
materiel,  leadership  and  education,  personnel,  and  facilities”  used  in  defense  planning  and  execution.  Materiel 
assets  are  items  such  as  weapons  systems,  equipment,  supplies.  For  mine  countermeasures  these  includes  ships, 
helicopters,  sonars,  and  marine  mammals. 

Non-materiel  items  such  as  doctrine,  of  course,  can  matter  greatly,  especially  in  war.  Military  historian 
Stephen  Ambrose,  for  example,  attributes  the  stiff  resistance  of  the  German  army  to  the  Allied  invasion  of 
France  in  1944  to  then-  superior  doctrine  and  training  in  fighting  in  the  hedgerows  of  Normandy4.  More  re¬ 
cently,  the  success  of  2008  surge  in  Iraq  can  be  attributed  not  only  to  an  increase  in  troop  strength  but  also  to 
General  Petraeus’  doctrine  of  “securing  the  population”  and,  more  particularly,  to  the  use  of  “fusion  cells”  of 
intelligence,  mapping,  and  forensic  specialists,  coupled  with  Joint  Special  Operations  Command,  to  locate  and 
target  al-Qaeda  operatives  [8].  In  a  non-pilot  effort,  analyses  of  both  materiel  and  non-materiel  alternatives  are 
essential. 

In  terms  of  costs,  the  pilot  effort  included  materiel  items  and  the  personnel  and  training  associated  with 
operating  and  supporting  them. 

4.2  Mine  Warfare 

To  understand  some  of  the  challenges  faced  by  the  Navy  team  in  actually  doing  portfolio  analysis,  it’s  worth¬ 
while  to  briefly  describe  the  fundamentals  of  mine  warfare. 

4.2.1  Mission  and  Threat 

The  mission  of  mine  warfare  is  to  deploy  mines  against  the  enemy  and  to  counter  mines  that  the  enemy  might 
deploy  against  coalition  forces. 

The  threat  of  enemy  sea  mines  to  our  maritime  forces  is  pervasive,  international,  and  asymmetric.  In 
addition  to  terrorist  groups,  more  than  50  countries  possess  sea  mines.  They  are  designed,  built,  and  sold  on 
the  international  market  from  friends,  second  parties,  and  potential  foes  alike.  Since  the  end  of  World  War  II, 
more  U.S.  Navy  ships  have  been  damaged  or  sunk  by  enemy  sea  mines  than  by  enemy  aircraft  and  missiles 
combined. 

Adversaries  can  deploy  mines  in  various  depth  regimes  ranging  from  deep  water  to  very-shallow  water  in 
the  littoral,  where  threat  is  particularly  acute  due  to  the  importance  of  coastal  areas  in  expeditionary  warfare 
and  to  the  enemy’s  ability  to  easily,  quickly,  and  cheaply  exploit  this  regime. 

4Facing  an  opponent  with  overwhelming  air  and  materiel  superiority,  the  Wehrmacht  nevertheless  managed  to  stymie  Operation 
Overlord  for  a  couple  of  months  [7], 
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Types  of  mines  include  floating,  moored,  tethered,  bottom,  buried,  rocket-propelled,  remotely  controlled, 
metallic,  and  stealthy. 

In  1988,  the  USS  Samuel  B.  Roberts  hit  an  Iranian  M-08  naval  mine  in  the  Persian  Gulf  while  escorting 
Kuwaiti  oil  tankers  during  the  Iran-Iraq  war.  The  mine’s  explosion  blew  a  15 -foot  hole  in  the  hull  and  ripped 
open  the  ship’s  engine  room.  The  crew  heroically  fought  flooding  and  flames  and  managed  to  prevent  the  ship 
from  sinking  [9]. 

She  was  repaired  at  Bath  Iron  Works  in  Maine  at  a  cost  of  $90  million  to  the  U.S.  Navy.  The  price  of  the 
mine  was  $1,500. 

While  the  Iranian  mine  was  based  on  World  War  II  technology,  designers  and  developers  of  mines  continue 
to  take  full  advantage  of  commercial  advances  in  electronics,  wireless  communication,  and  nanotechnology  to 
make  their  products  more  stealthy  and  lethal  than  ever. 

4.2.2  Portfolio 

Figure  2  depicts  the  threat  of  sea  mines  to  U.S.  naval  and  coalition  maritime  forces,  and  the  portfolio  of  current 
and  planned  countermeasures. 

Organic  Navy  Assets 
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Five  new  systems 

UUVs;  MRUUV 
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SYQ-13;  MEDAL 
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Figure  2:  The  threat  of  sea  mines 

Focusing  first  on  threat,  the  diversity  of  depth  regimes  and  the  variety  of  mines  within  each  pose  a  major 
challenge  to  our  operating  forces,  and  demand  a  variety  of  search  and  neutralization  techniques.  In  mine 
warfare,  there’s  no  silver  bullet  or  simple  solution.  One  system  does  not  do  all.  Instead,  countering  enemy  sea 
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mines  is  regarded  as  one  of  the  most  complex  and  difficult  realms  of  naval  warfare.  Multiple  systems  address 
the  kill-chain  steps  of  search  (look  for  objects),  locate  (specify  their  position  in  the  water,  ocean  bottom,  or 
beach),  classify  (bin  them  into  mine-like  and  not-mine-like  categories),  identify  (determine  which  mine-like 
objects  are,  in  fact,  mines),  and  neutralize  (render  the  mines  harmless). 

Focusing  now  on  counter  measures,  at  the  heart  of  mine  warfare  is  intelligence,  surveillance,  and  reconnais¬ 
sance.  This  is  where  many  of  our  national,  strategic  assets  play  a  role.  These  include  the  National  Reconnais¬ 
sance  Office’s  constellations  of  polar-orbiting  satellites  with  remote  sensing  capability;  the  National  Geospatial 
Intelligence  Agency’s  digital  maps  of  the  littoral;  and  the  Defense  Intelligence  Agency’s  human,  ashore  assets 
which  many  analysts  think  will  be  increasingly  important  in  the  decades  ahead. 

At  the  operational  level,  the  U.S.  Navy  uses  dedicated  and  organic  assets.  The  older,  traditional  dedicated 
systems  are  devoted  fully  or  largely  to  mine  warfare.  The  newer  organic  assets,  on  the  other  hand,  travel 
with  Carrier  Strike  Groups  and  attempt  to  meet  the  Navy’s  long-standing  goal  of  removing  the  sailor  from  the 
minefield. 

Dedicated  assets  are  a  triad  of  surface,  airborne,  and  undersea  systems.  Included  are  14  Avenger-class 
ships  built  in  the  1980s  and  early  1990s  of  fiberglass-sheathed  wooden  hulls  that  enable  them  to  operate  within 
minefields.  They  deploy  long  cables  astern  charged  with  magnetic  and  acoustical  signals  that  emulate  those  of 
our  ships  and  submarines  in  an  attempt  to  induce  enemy  sea  mines  into  detonating.  The  vessels  also  deploy  the 
tethered  SQQ-32  sonar  for  mine  detection  and  classification  and  the  SLQ-48  for  subsequent  mine  identification 
and  neutralization. 

The  multi-mission  MH-53E  helicopter  is  the  largest  in  DoD’s  arsenal.  The  Sea  Dragon  tows  mine-hunting 
sonar  such  as  the  “Q-20”  and  the  newly  deployed  “Q-20A”  to  find  sea  mines  and  mechanical,  magnetic,  and 
acoustic  sweep  gear  to  neutralize  them. 

The  Explosive  Ordnance  Disposal  units  have  the  daunting  mission  of  finding  and  disarming  mines.  Navy 
divers  work  closely  with  dolphins  and  sea  lions  who  have  a  unique  ability  to  find  and  neutralize  mines  nestled 
in  the  ocean  sands. 

Turning  to  the  organic  domain,  the  Littoral  Combat  Ship  (LCS),  along  with  its  Fire  Scout  unmanned  aerial 
vehicle5,  will  be  the  U.S.  Navy’s  mine  countermeasure  platform  of  the  future.  The  USS  Freedom,  the  first  ship 
in  the  class,  was  delivered  to  the  Navy  in  2008.  The  multi-mission  vessels  will  be  configurable  to  perform  mine, 
anti-submarine,  and  surface  warfare  missions.  LCS  will  transport  manned  and  unmanned  mine  countermeasure 
systems  to  the  littoral,  where  they  will  be  deployed,  while  the  ship  remains  offshore  and  out  of  harm’s  way. 

The  MH-60S  Knighthawk  helicopter  will  deploy  from  the  ship,  and  will  cany  a  number  of  new  mine 
countermeasure  systems  which  have  emerged  from  the  science  and  technology  community  and  are  currently  in 
development  or  testing. 

These  include  the  Airborne  Mine  Neutralization  System  and  the  Rapid  Airborne  Mine  Clearance  System 
which  rapidly  fires  30mm  tungsten  projectiles  from  a  Gatling-type  gun  into  the  water  to  destroy  enemy  sea 
mines  by  sheer  kinetic  force. 

Additional  assets  in  the  mine  warfare  portfolio  include  unmanned,  undersea  vehicles  (UUVs),  command 
and  control  systems,  and  assault  systems  for  mine  clearance  in  the  surf  zone  and  craft  landing  zone.  Finally,  the 
naval  research  community  continues  to  develop  new  technologies  for  countering  enemy  sea  mines,  especially 
in  the  littoral. 

The  U.S.  Navy  pilot  effort  excluded  the  national,  strategic  assets  from  consideration.  These  are  classified 
and  devote  but  a  small  percentage  of  their  time  to  mine  warfare.  This  left  46  different  systems  plus  a  couple  of 
classes  of  ships  and  helicopters  to  analyze  in  the  portfolio  of  mine  countermeasures. 

5  In  its  mine  countermeasure  mode,  the  Fire  Scout  will  be  used  for  a  military  task  called  “intelligence  preparation  of  the  battlefield.” 
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4.3  Methodology 

4.3.1  Assessing  Capability  -  System  Architecture 

The  naval  task  group,  with  strong  representation  from  the  mine  warfare  community,  met  monthly  during  much 
of  2005  to  brainstorm  ideas  for  assessing  the  military  value  of  systems  in  the  portfolio,  both  current  and  prospec¬ 
tive,  and  large  and  small  alike6.  The  task  group  recognized  immediately  the  necessity  of  tying  value  to  national 
security  goals,  set  by  higher  authority.  It  wanted  to  avoid  the  error  of  operating  like  loose  cannon  on  the  decks 
of  the  Department  of  Defense. 

The  team  received  invaluable  advice  and  support  from  professors  and  staff  at  the  Naval  War  College  in  New¬ 
port,  Rhode  Island,  on  the  business  of  scoring  the  importance  of  objectives.  Following  the  lead  of  USSOCOM, 
the  team  developed  a  strategy-to-systems  model  for  assessing  the  military  value  of  systems  in  the  portfolio. 

As  a  backdrop  to  exercising  the  model,  illustrated  in  Figure  3,  the  team  first  defined  a  master  warfighting 
scenario.  This  was  an  amalgam  of  two  separate  scenarios,  one  for  a  major  combat  operation  overseas  and 
one  for  homeland  defense,  thus  giving  each  of  the  assets  in  the  mine  countermeasure  portfolio  a  chance  to 
demonstrate  its  true  capabilities.  While  unclassified,  the  master  scenario  nevertheless  detailed  factors  such  as 
bottom  profile,  tactical  situation,  mine  threat,  and  dimensions  and  water  depth  of  sea  lines  of  communication. 


Identification 
and  Ranking 
of  Strategic 
Tasks 


Weightings 


Operational  to  Strategic  Tasks 


^  Weighings 
Tactical  to  Operational  Tasks 


1 


Weightings 


Systems  to  Tactical  Tasks 
—  Effectiveness 
-Risk 


J ^ 


Figure  3:  The  master  warfighting  scenario 

Next,  the  team  identified  and  then  scored  the  importance  of  five  strategic  tasks  or  requirements  for  mine 
countermeasures.  These  tasks  were  tied  to  National  Defense  Strategy  and  recent  Presidential  directives  for 
increasing  security  in  the  maritime  domain  [10]. 

The  team  then  defined  six  operational  tasks  and  scored  their  importance  in  fulfilling  the  five  strategic  re¬ 
quirements.  The  team  similarly  defined  12  tactical  tasks  and  related  these  to  operational  tasks.  Finally,  the  team 
graded  individual  systems  against  tactical  tasks. 

6Task  group  members  included  representatives  from  Mine  Warfare  Command;  Naval  Surface  Warfare  Center  Panama  City;  Naval 
Undersea  Warfare  Center;  Naval  War  College  (Warfare  Analysis  and  Research  and  the  Decision  Support  Center);  Mine  Warfare  Program 
Office  (PMS-495);  the  Chief  of  Naval  Operations  (Expeditionary  Warfare  and  Mine  Warfare);  and  the  Office  of  Naval  Research. 
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This  linkage  of  strategy  to  tasks  to  systems  was  deliberate.  In  this  sort  of  schema,  those  systems  that 
score  highest,  ceteris  paribus,  are  those  that  best  respond  to  the  strategic  goals  articulated  by  the  President,  the 
Secretary  of  Defense,  and  the  Joint  Chiefs  of  Staff. 

4.3.2  Assessing  Capability  -  Execution  Details 

The  team  conducted  a  scoring  conference  at  the  Decision  Support  Center  at  the  U.S.  Naval  War  College  in 
December  2005.  It  employed  a  “Borda  Count”  technique  to  measure  the  rank  order  and  relative  importance  of 
the  five  strategic  tasks,  shown  in  Table  1,  while  recognizing  that  no  flawless  procedure  exists  for  doing  so.  As 
Arrow’s  Impossibility  Theorem  indicates,  all  techniques  to  rank-order  preferences,  other  than  using  a  dictator, 
will  violate  at  least  one  commonly  accepted  measure  of  fairness  [11]. 


Table  1 :  Scoring  of  strategic  requirements 


Strategic  Requirements 

Sample  Score 

Protect  Operating  Forces  Against  the  Threat  of  Sea  Mines  for  Power  Projection  Ashore 

15 

Defend  U.S.  Ports  and  Coastal  Approaches  Against  Sea  Mines 

50 

Maintain  Mobility  of  Operational  Forces  in  the  Presence  of  Sea  Mines 

13 

Collect,  Analyze,  and  Share  Intelligence  Related  to  the  Worldwide  Threat  of  Sea  Mines 

10 

Preserve  Freedom  of  the  Seas  for  Commercial  Navigation  in  the  Presence  of  Sea  Mines 

12 

Each  scorer  was  given  a  total  of  100  points  to  distribute  among  the  five  requirements7.  While  the  numbers 
here  are  fictitious,  “Defend  U.S.  Ports  . . .”  did  emerge  as  the  top-rated  priority  in  our  scoring  conference.  This 
is  perhaps  not  surprising  upon  noting  that  U.S.  ports  load  and  unload  from  ships  an  average  of  7 1 ,000  containers 
daily  [13]  and  that  the  value  of  commerce  at  the  Port  of  Long  Beach  alone  is  $100  billion  per  year  [14], 

The  team  scored  the  importance  of  each  of  the  six  operational  tasks  in  meeting  each  of  the  five  strategic 
requirements  using  a  template,  shown  in  Figure  4,  developed  using  a  web-based  decision  analysis  support  tool8. 

After  scoring  the  importance  of  tactical  tasks  in  fulfilling  operational  tasks,  the  grouped  the  46  individual 
mine  countermeasure  systems  into  these  categories  according  to  their  basic  function: 

•  Detection,  localization,  classification,  and  identification  of  sea  mines 

•  Neutralization  of  sea  mines 

•  Mine  sweeping 

•  Assault 

7This  technique  avoids  the  pitfall  of  cardinal  ordering  by  measuring  the  amount  by  which  one  requirement  is  judged  more  important 
than  another  [12], 

8The  six  operational  tasks  and  the  12  tactical  tasks  (not  displayed  here)  were  based  largely  on  a  distillation  of  guidance  provided  in 
the  "Universal  Joint  Task  List"  from  the  Joint  Chiefs  of  Staff  and  from  similar  U.S.  Navy  guidance.  The  guidance  lists  military  tasks  or 
steps  that  need  to  be  executed  to  accomplish  a  mission,  such  as  countering  enemy  seas. 
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Figure  4:  Sample  scoring  template 


•  Command  and  control. 

The  team  then  rated  the  effectiveness  and  suitability  of  each  system,  tackling  one  category  of  assets  at  a 
time,  in  executing  tactical  tasks.  This  was  to  keep  the  scoring  manageable  and  to  help  insure  consistency  in 
evaluating  systems  with  the  same  mission.  Effectiveness  was  evaluated  in  terms  of  the  overall  ability  of  a 
system  to  accomplish  its  mission.  Suitability  was  evaluated  in  terms  of  degree  of  reliability,  maintainability, 
and  interoperability  of  the  asset  in  a  warfighting  environment. 

Aggregate  capability  scores  were  tallied  for  each  system  using  the  cascading  series  of  weights  from  the 
rankings  of  strategic,  operational,  and  tactical  tasks,  applied  to  the  numerical  evaluations  of  effectiveness  and 
suitability.  The  team  converted  raw  scores  into  index  numbers,  using  1 .0  as  the  mean  or  base  value  for  the  entire 
portfolio. 

4.3.3  Estimating  Risks 

The  team  assessed  risk  according  to  a  system’s  phase  in  its  life  cycle: 

•  Science  and  technology  projects 

Risk  of  failing  to  meet  design,  research,  and  transition  goals; 

•  Acquisition  programs 

Degree  of  design,  platform  integration,  and  testing  challenges; 

•  Operational  systems 

Risk  of  failing  to  meet  mission  requirements. 

In  addition,  the  team  developed  risk  weighting  factors  for  each  phase,  running  from  high  to  low.  An  S&T 
project,  for  example,  carries  inherently  greater  risk  than  an  operational  system,  on  average,  since  it  first  needs  to 
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achieve  research  goals  and  then  pass  through  acquisition  successfully  before  becoming  a  useful  military  asset. 
This  is  not  to  say  that  operational  systems  are  without  risk.  Dolphins  and  sea  lions,  for  example,  are  sometimes 
described  as  “capricious”  in  hunting  for  sea  mines.  When  they  want  to  work,  they  do  a  great  job.  Some  days, 
however,  they  prefer  to  play. 

4.3.4  Estimating  Costs 

Measuring  military  value  and  risk  of  46  systems  and  four  platforms  was  only  half  the  work  of  the  pilot  program. 
Given  the  scarcity  of  resources  allocated  to  national  security  in  a  pluralistic,  free  society,  it  was  essential  to  scale 
capability  by  cost  to  provide  a  “value  for  money”  metric,  to  use  a  favorite  phrase  of  British  defense  planners. 
Team  members  believed  strongly  in  the  criticality  of  capturing  the  total  costs  of  assets  over  their  service  life 
and  not  simply  acquisition  costs  or  costs  in  the  year  of  the  budget9.  Depending  upon  where  a  system  was  in 
its  life-cycle,  this  required  estimates  of  development,  production,  and  operating  and  support  costs,  as  Figure  5 
shows. 


Figure  5:  Life  cycle  of  defense  systems 

The  job  of  cost  estimation  in  the  mine  countermeasure  portfolio  was  made  more  difficult  because  of  the 
numbers  of  relatively  small,  inexpensive  systems  involved.  Many  of  these  eluded  the  minimum  dollar  threshold 
for  cost  reporting  in  the  U.S.  DoD,  and  therefore  required  a  variety  of  estimation  techniques. 

4.4  Findings 

Combining  assessments  of  military  capability  and  life-cycle  cost  analysis  for  the  mine  countermeasure  systems, 
the  naval  task  group  found  risk-reward  bubble  diagrams  particularly  useful  in  displaying  results.  Figure  6  is 

9In  defense  cost  analysis,  operating  and  support  costs  are  usually  estimated  for  ten  years  after  initial  operational  capability  (IOC)  of 
a  system,  or  the  date  of  fielding.  For  mine  countermeasure  systems,  this  often  required  estimation  of  costs  through  2020. 


RTO-TR-SAS-076 


C  -13 


ANNEX  C  -  THE  ROLE  OF  LIFE  CYCLE  COST 
ANALYSIS  IN  MANAGING  THE  DEFENSE  ENTERPRISE 


ORGANIZATION 


an  example  for  the  16  systems  in  the  detect-to-identify-sea- mines  category.  The  size  of  each  bubble  repre¬ 
sents  resources  to  be  expended,  or  estimated  life-cycle  costs,  in  2006  constant  dollars,  scaled  to  a  reference 
point  of  $100M.  Costs  include  development,  production,  and  operating  and  support,  where  applicable.  Each 
accompanying  number  is  an  estimated  return  on  investment  (ROI),  or  military  value  divided  by  bubble  size. 

ROI  =  Military  Value/Bubble  Size 
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Figure  6:  Detection,  localization,  classification,  and  identification  of  sea  mines 

The  two  axes  intersect  at  coordinates  (1.0,  1.0),  the  global  portfolio  average  values  for  risk  and  reward  - 
measured  across  all  46  systems. 

The  team  labeled  the  four  quadrants  using  nautical  terminology.  The  BZs,  or  Bravo  Zulus,  meaning  “Well 
Done”,  arc  the  potential  stars,  the  systems  with  high  return  and  low  risk.  Moving  in  counter-clockwise  direc¬ 
tion,  the  Oysters  are  long-shot  programs.  These  are  often  projects  that  could  yield  a  high  military  return  if 
technological  breakthroughs  are  achieved.  Moving  along,  every  organization  seems  to  have  its  share  of  bilge 
water,  or  at  least  one  project  or  system  of  low  value  that’s  hard  to  kill.  Finally,  the  watch  standing  systems 
are  the  no-brainers.  These  have  a  high  likelihood  of  success  but  a  low  or  moderate  return.  They  often  involve 
modifications,  extensions,  and  fixes  to  older  systems. 

A  couple  of  observations  are  readily  apparent  from  the  display.  First,  the  detect-to-identify  systems  do  very 
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well  as  a  group.  Roughly  2/3  of  them  fall  to  the  right  of  the  global  mean.  This  is  because  of  the  numbers  and 
importance  of  the  tactical,  operational,  and  strategic  tasks  they  support.  In  mine  warfare,  it's  all-important  to 
find  the  mines  in  the  first  place  regardless  of  mission.  U.S.  forces  sometimes  have  the  opportunity  of  going  over 
or  around  sea  mines,  at  least  in  the  short  run. 

Second,  there  are  profound  differences  in  ROIs.  An  examination  of  the  underlying  data  reveals  that  the 
deltas  are  driven  mostly  by  differences  in  cost  rather  than  effectiveness.  The  off-the-chart  outlier  with  the  ROI 
of  80,  by  the  way,  is  a  relatively  inexpensive  unmanned,  underwater  vehicle  designed  to  engage  enemy  sea 
mines  in  the  littoral.  Principal  decision  makers  might  logically  question  the  acquisition  of  the  more  expensive 
systems.  This  kind  of  insight  is  a  potential  payoff  of  portfolio  analysis. 

Balance  in  the  portfolio  can  be  assessed  by  evaluating  the  mix  of  S&T,  acquisition,  and  operational  systems, 
and  their  risks  and  returns,  within  and  between  categories  of  assets.  Interestingly,  to  take  one  example,  S&T 
projects  represent  roughly  10%  of  the  total  number  of  systems  in  the  detect-to-identify  category  but  20%  in  the 
mine-sweeping  category.  But,  as  Figure  7  shows,  sweeping  systems  perform  poorly  as  a  group  because  of  the 
paucity  of  tasks  they  support.  Decision  makers  might  logically  question  if  the  percentage  allocations  shouldn’t 
be  reversed  based  on  the  criterion  of  relative  worth. 


Figure  7:  Analysis  of  detection,  localization,  classification,  and  identification  of  sea  mines 

Strategic  fit  is  measured  implicitly  by  the  numerical  military  value  of  a  system.  The  identification  and  rank¬ 
ing  of  strategic  requirements  is  the  lynchpin  of  the  entire  analysis.  Weights  assigned  to  strategic  requirements 
cascade  to  supporting  levels  of  tasks  and  systems.  Any  countermeasure  asset  that  doesn’t  adequately  support 
strategy  is  simply  not  going  to  score  well.  The  lowest-rated  system  in  Figure  7,  for  example,  is  an  S&T  project 
that  appears  “off-strategy”,  at  least  at  first  blush.  Additional  investigation  seems  warranted  into  the  merits  of 
the  project. 
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In  addition  to  assessing  the  capability  of  individual  systems,  the  task  group  analyzed  the  military  value  of 
groups  of  interrelated  systems.  For  example,  as  Figure  8  shows,  a  number  of  mine  countermeasure  systems 
are  deployed  from  the  legacy  MH-53E  helicopter  and  from  the  new  MH-60S  helicopter,  with  both  labeled 
“platforms”  in  the  display. 


Figure  8:  Analysis  of  groups  of  interrelated  systems:  MH-53E  and  MH-60S  helicopters 

One  helicopter  scores  higher  in  capability  than  the  other  but  their  life-cycle  costs  are  roughly  the  same. 
The  MH-60S  is  a  new  system  now  beginning  built.  Interesting,  its  production  costs  are  offset  by  operating  and 
support  costs  which  are  much  lower  than  the  expensive-to-operate  and  aging  MH-53E.  In  terms  of  the  systems 
deployed  from  these  platforms,  the  new  MCM  systems,  or  those  carried  by  the  new  helicopter,  do  not  show 
a  marked  advantage  in  capability  over  the  older  legacy  systems.  Indeed,  a  couple  of  advanced  sonar  systems 
earned  by  the  legacy  helicopter  yielded  the  most  military  value  in  our  pilot  analysis. 

4.5  SAS-076  Evaluation 

The  SAS-076  Task  Group  critically  evaluated  the  U.S.  pilot  effort  in  capability  portfolio  analysis  and  offers 
these  comments: 
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•  The  methodology  of  the  pilot  effort  seems  appropriate  for  complementing  rather  than  duplicating  existing 
decision-support  systems  and  processes  within  the  ministries  of  defense  in  NATO  and  PfP  nations.  The 
pilot’s  requirements-to-systems  model,  for  example,  does  not  identify  capability  gaps. 

•  The  pilot  effort  does  not  adequately  address  the  dimensions  of  capacity  and  time.  Identifying  the  number 
of  units  required,  for  each  kind  of  asset,  is  critical.  Further,  some  units  may  be  required  before  others,  all 
have  limited  service  lives,  and  threats  are  often  highly  dynamic  and  change  over  time. 

•  Use  of  scorers  is  probably  inevitable;  however,  subjectivity  should  be  minimized  and  known  performance 
characteristics  of  equipment  better  utilized. 

•  Nevertheless,  the  pilot  effort  seems  to  demonstrate  that  portfolio  analysis  techniques  can  indeed  be  em¬ 
ployed  fruitfully  in  national  defense.  In  short,  the  pilot  effort  in  mine  countermeasures  should  be  regarded 
as  a  couple  of  steps  in  the  right  direction  in  providing  senior  defense  leadership  with  a  tool  that  measures 
the  costs,  capabilities,  and  risks  of  individual  systems  while  linking  them  to  national  security  require¬ 
ments. 

4.6  SAS-076  Recommendations 

SAS-076  recognizes  that  no  model  is  perfect.  As  the  Dutch  Econometrician  Henri  Theil  once  famously  said, 
“models  are  to  be  used,  but  not  believed.”  [15].  That  is,  models  are  parsimonious,  plausible,  and  hopefully 
informative,  but  do  not  provide  the  ultimate  truth.  Nevertheless,  SAS-076  makes  these  recommendations  re¬ 
garding  future  NATO  efforts  in  the  area  of  capability  portfolio  analysis: 

•  Include  a  menu  of  portfolios  from  which  to  choose, 

•  Include  dimensions  of  capacity  and  time, 

•  Include  estimates  of  life-cycle  costs, 

•  Present  joint  solution  space,  and 

•  Minimize  subjectivity  in  assessing  military  value. 

On  the  first  count,  the  current  portfolio  or  program  of  record  for  mine  countermeasures  was  merely  one 
manifestation  of  reality.  Other  portfolios  are  imaginable.  “High-tech”,  for  example,  would  emphasize  science 
and  technology  projects.  A  “coalition”  portfolio,  on  the  other  hand,  might  include  the  assets  from  a  NATO 
Alliance  point  of  view,  with  many  nations  having  a  robust  mine  countermeasure  capability10.  This  addresses 
another  point,  or  the  need  to  view  joint  solutions,  not  only  among  U.S.  DoD  components  but  internationally  as 
well. 

On  the  issue  of  subjectivity,  a  key  metric  in  mine  warfare  is  ocean  coverage  per  unit  of  time.  This  value 
is  known  within  fairly  tight  bounds  for  many  countermeasure  systems  and  can  be  used  in  follow-on  analysis. 
However,  some  use  of  opinion  is  inevitable  since  warfare  is  both  art  and  science.  In  short,  we  need  “a  nexus  of 
academic  rigor  and  operator  experience”. 


1()The  Netherlands,  for  example,  has  a  fleet  of  10  mine  warfare  ships  [16]. 
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5.0  NATIONAL  TEMPLATES 

5.1  Overview 

Completed  templates  or  presentation  slides  were  received  from:  Canada,  Turkiye,  United  States,  Sweden, 
Norway,  Germany,  and  the  Netherlands.  Each,  in  one  fashion  or  another,  attempted  to  present  details  on: 

•  Overview  of  management  of  the  defense  enterprise; 

•  Strategic  framework; 

•  Needs  and  solutions; 

•  Lexicon  and  taxonomy;  and 

•  Role  of  life-cycle  cost  analysis. 

5.2  Commonalities  and  Differences 

The  SAS-076  Task  Group  recognizes  that  every  nation  within  the  Alliance,  and  among  PfP  nations,  has  its 
own  procedures  and  processes  for  establishing  strategic  guidance  and  objectives;  for  identifying  military  needs; 
and  for  developing  and  procuring  solutions.  The  application  of  life-cycle  cost  analysis,  captured  in  the  fifth 
template,  likely  therefore  differs  among  nations,  too,  according  to  the  scope  and  complexity  of  its  planning 
and  acquisition  processes.  The  SAS-076  Task  Group  identified  the  following  commonalities  and  differences  in 
national  practices  from  the  nations  reporting. 

5.3  Strategic  Framework 
Areas  of  Analysis 

•  Goal  setting 

•  Global  analysis 

•  National  defense  analysis 

•  Strategy  formulation 

•  Testing  the  strategy 


Summary 

All  nations  in  the  sample  state  general  goals  or  ambitions  in  a  national  security  document.  Some  nations  offer 
more  detail  than  others.  For  example,  some  nations  proffer  ambitions  in  a  worldwide  context  while  others  state 
goals  in  the  context  of  specific  scenarios.  All  nations  have  processes  and  products  that  translate  high-level 
objectives  into  more  concrete  policy  parameters  needed  to  build  military  forces.  Most  nations  in  the  sample 
and  many  others  within  NATO  engage  in  capability-based  planning. 


C  - 18 


RTO-TR-SAS-076 


ANNEX  C  -  THE  ROLE  OF  LIFE  CYCLE  COST 
ANALYSIS  IN  MANAGING  THE  DEFENSE  ENTERPRISE 


Occasion  a  Mot  Concrete 

|  Hno 

|  ?  |  Response  Unclear 


U.S.  NLD  CAN  SWE  NOR  TUR 

Goal  Setting 

Stated  officially  in  a  document 
Periodicity  of  publication  (x  years) 

Presented  by  Ministry  of  Defense  or  equivalent 
Global  Analysis 

Described  in  an  offical  document 
Periodicity  of  global  anlaysis  publication  (x  years) 

Looking  ahead  (Y)  or  Static  (N) 

Period  considered  for  global  trends  analysis  (x  years) 

National  Defense  Analysis 

Described  in  an  offical  document 

Periodicity  of  national  defense  anlaysis  publication  (x  years) 

Looking  ahead  (Y)  or  Static  (N) 

Period  considered  for  national  defense  trends  analysis  (x 
years) 

Integration  of  defense  multinational  cooperation 
Application  of  capability  portfolio  analysis 

Strategy  Formulation 

Described  in  an  offical  document 
Periodicity  of  coverage  (x  years) 

Describes  possible  scenarios  &  reasons  for  choice  of 

Includes  considerations  on  multinational  cooperation 

Testing  the  Strategy 


QDR 

Policg  View 

National  Securitg 
Strategg 

Defense 
Policg  Bill 

Defense 
Policg  Targets 

Turkish 

National 

! _ i _ 

4 

»  1  ’ 

DoD 

MoD 

MoD 

Government 

MoD 

MoD  •  General 
Staff  of  TAF 

1 

1 - 1 

Policg  View 

National  Securitg 
Strategg 

Defense 
Policg  Bill 

7 

Turkish 

National 

? 

3 

7 

? 

Scenarios 

7 

7 

? 

1 _ » _ 

7 

7 

7 

7 

Policg  View 

Policg 

Recommendations 

Defense 
Policg  Bill 

Strategic 

Analgsis 

TAF 

Operational 

Requirements 

Scenarios 

Policg  Options 

Mid-term 

Figure  9:  Summary  of  analysis  of  nations’  responses  with  respect  to  strategic  planning. 


United  States 

National  Security  Strategy,  signed  by  the  President  of  the  United  States,  provides  top-level  strategic  direction 
and  is  required  by  law  after  a  Presidential  election.  The  Quadrennial  Defense  Review  is  mandated  by  law  and 
requires  DoD  to  undertake  a  comprehensive  examination  of  its  strategy  and  performance. 

“The  Secretary  of  Defense  shall  every  four  years ...  conduct  a  comprehensive  examination  [the 
QDR ]  of  the  national  defense  strategy,  force  structure,  force  modernization  plans,  infrastructure, 
budget  plan,  and  other  elements  of  the  defense  program  and  policies  of  the  United  States  with  a 
view  toward  determining  and  expressing  defense  strategy  of  the  United  States  and  establishing  a 
defense  program  for  the  next  20  years.  Each  such  [QDR]  shall  be  conducted  in  consultation  with 
the  Chairman  of  the  Joint  Chiefs  of  Staff.”  [United  States  Code  10]. 

To  date,  QDRs  have  been  conducted  in  1997,  2001,  and  2005,  and  2009,  and  each  of  those  reviews  has 
resulted  in  substantial  changes  to  defense  strategy.  The  trend  has  been  that  the  QDR  report  is  the  first  document 
that  details  an  incoming  Administration’s  views  on  national  defense,  and  it  generally  leads  the  production 
of  other  DoD  strategic  guidance.  National  Defense  Strategy,  signed  by  the  Secretary  of  Defense,  contains 
guidance  on  security  challenges,  key  operational  capabilities,  and  operational  priorities.  National  Military 
Strategy,  signed  by  the  Chairman  of  the  Joint  Chiefs  of  Staff  (CJCS),  provides  operational  context  to  the  Defense 
Strategy.  Guidance  for  Employing  the  Force  says  how  to  employ  the  force  in  terms  of  operational  plans, 
global  force  posture,  and  aims  for  security  cooperation.  Guidance  for  Developing  the  Force,  on  the  other  hand, 
contains  program  guidance;  that  is,  what  capabilities  should  be  fielded  and  when.  Both  of  these  documents  are 
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ORGANIZATION 


supported  by  more  detailed  implementing  guidance. 

Canada 

The  Government  of  Canada’s  national  security  strategy  is  typically  outlined  in  several  key  policy  documents, 
including  Speeches  from  the  Throne,  major  speeches  by  the  Prime  Minister,  the  annual  Budget,  statements 
by  Ministers  in  Parliament,  and  Government  announcements.  While  there  is  currently  no  updated  overarch¬ 
ing  national  security  strategy  document,  Public  Safety  Canada  (PS)  is  the  federal  department  responsible  for 
leading  the  development  and  implementation  of  Canada’s  national  security  strategy.  PS  works  with  five  agen¬ 
cies,  including  the  Canada  Border  Services  Agency,  the  Canadian  Security  Intelligence  Service,  and  the  Royal 
Canadian  Mounted  Police,  and  unites  them  under  a  single  portfolio  and  minister.  The  result  is  better  inte¬ 
gration  among  federal  organizations  dealing  with  national  security,  emergency  management,  law  enforcement, 
corrections,  crime  prevention  and  borders.  In  addition  to  coordinating  and  supporting  the  efforts  of  federal 
organizations  to  ensure  national  security,  PS  also  works  with  other  levels  of  government,  first  responders,  com¬ 
munity  groups,  the  private  sector  and  other  nations. 

A  key  part  of  the  Government’s  national  security  strategy  is  Canada’s  defense  policy.  When  a  new  Gov¬ 
ernment  takes  office,  it  will  typically  set  out  its  defense  policy  objectives  and  its  corresponding  direction  to 
the  Department  of  National  Defence  and  the  Canadian  Forces  (DND/CF).  The  process  for  developing  defense 
policy  is  flexible  and  varies,  but  always  requires  departmental  and  political  involvement. 

Canadian  defense  policy  development  and  planning  involves  analyzing  the  national  and  international  strate¬ 
gic  environment,  the  defense  and  security  needs  of  the  country,  and  the  Government’s  agenda,  priorities,  and 
fiscal  context.  From  this,  the  required  military  capabilities  are  identified  and  policy  options  are  formulated. 
Working  with  senior  military  leadership,  the  Assistant  Deputy  Minister  (Policy)  within  DND/CF  is  responsible 
for  developing  policy  options,  and  for  coordinating  with  other  departments  and  agencies  (e.g.  Department  of 
Foreign  Affairs  and  International  Trade;  Privy  Council  Office)  to  ensure  government-wide  policy  coherence. 
Consultation  with  others  external  to  the  government  may  also  occur,  both  with  national  groups  (e.g.  Parliamen¬ 
tary  committees  on  defense  and  security;  academia)  and  international  actors  (e.g.  NATO;  partner  countries). 
Once  the  Deputy  Minister  and  the  Chief  of  the  Defence  Staff  agree  on  the  content  of  the  proposals,  policy 
options  are  discussed  with  the  Minister  of  National  Defence,  who  is  then  responsible  for  presenting  recommen¬ 
dations  to  Cabinet  and  the  Prime  Minister. 

The  Canada  First  Defence  Strategy  sets  a  detailed  road  map  for  the  modernization  of  the  Canadian  Forces. 
It  puts  forward  clear  roles  and  missions  for  the  Canadian  Forces,  outlining  a  level  of  ambition  that  will  enable 
the  CF  to  maintain  the  ability  to  deliver  excellence  at  home,  be  a  strong  and  reliable  partner  in  the  defense 
of  North  America,  and  project  leadership  abroad  by  making  meaningful  contributions  to  operations  overseas. 
Canada  First  Defence  Strategy  is  the  road  map  for  the  modernization  of  the  Canadian  Forces  and  is  based  on 
analysis  of  the  risks  and  threats  facing  Canada  in  the  years  to  come,  as  well  as  the  government’s  vision  for 
defense.  The  strategic  decision-making  process  for  defense  acquisitions  is  called  the  Force  Development  (FD) 
process  and  the  output  is  the  Investment  Plan  providing  detailed  guidance  on  how  and  when  to  invest. 

The  Netherlands 

The  new  Dutch  national  security  planning  consists  of  three  new  major  components  in  addition  to  the  existing 
planning  mechanisms  within  the  departments: 

1.  A  new  government-wide  (meta-)  foresight  function,  feeding 

2.  A  national  risk  assessment  which,  in  turn,  forms  the  basis  for 
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3.  A  strategic  planning  system  for  national  security,  based  on  government-wide  capabilities-based  planning. 
Future  Policy  Survey 

In  order  to  develop  an  adequate  conceptual  foundation  for  decisions  that  will  be  necessary  with  respect  to  the 
future  of  the  Armed  Forces,  the  Netherlands  government  initiated  the  Future  Policy  Survey.  The  Survey  makes 
a  substantial  and  scientifically  sound  contribution  to  political  and  public  perceptions  about  the  future  of  the 
Armed  Forces.  The  Future  Policy  Survey  is  an  expression  of  the  political  will  to  do  justice,  now  and  in  the 
future,  to  the  requirement  that  the  Armed  Forces  serve  as  a  crucial  safeguard  against  threats  to  the  Netherlands 
and  their  society. 

The  Future  Policy  Survey  project  began  on  1  March  2008  based  on  the  plan  of  action  approved  by  the 
Netherlands  defence  and  on  the  terms  of  reference  stated  in  the  plan.  Its  assignment  was:  “To  formulate,  on  the 
basis  of  expected  long-term  developments  and  possible  scenarios  and  without  constraints,  policy  options  with 
regard  to  the  future  ambitions  of  the  Netherlands  defence  effort,  the  appropriate  composition  and  equipment 
of  the  armed  forces,  and  the  associated  level  of  defence  expenditure.”(House  of  Representatives,  2008  31  243, 
No.  6).  The  Future  Policy  Survey  makes  a  scientifically  sound  contribution  to  political  and  public  perceptions 
about  the  future  of  the  Armed  Forces.  Among  other  things  The  Future  Policy  Survey  has  resulted  in  the  devel¬ 
opment  of  future  scenarios  that  set  out  the  main  outlines  of  how  the  world  may  develop,  and  the  consequences 
that  those  developments  could  bring  in  the  coming  two  decades.  The  scenarios  have  been  used  as  a  touchstone 
for  the  development  of  policy  options.  The  policy  options  have  been  used  to  determine  the  level  of  Dutch 
defense  efforts  over  the  long  term.  The  results  of  the  Future  Policy  View  have  been  used  as  an  input  for  the 
regular  Policy,  Planning  and  Budgeting  (BPB-Dutch)  procedure.  The  BPB-procedure  contains  a  policy  view 
and  a  defense  plan.  The  policy  view  is  the  first  decision-making  document  in  a  4-year  cycle  of  government 
and  the  document  determines  in  outlines  the  ambition  level  and  the  composition  and  equipment  of  the  armed 
forces.  In  the  defense  plan  the  policy  view  is  elaborated  in  more  detail  and  results  in  the  required  composition 
and  equipment  of  the  armed  forces.  The  objectives  are  detailed  in  activities  and  required  means.  The  defense 
plan  contains  proposals  for  the  design  of  the  organization.  The  defense  plan  has  a  horizon  of  10  years,  but  it 
includes  also  topics  with  a  longer  timeline. 

Norway 

The  Norwegian  Ministry  of  Defence  (NMoD)  uses  Strategic  Defence  Reviews  (SDR)  in  the  overall  planning  of 
the  defence  sector.  In  the  SDR  you  conduct  a  comprehensive  examination  of  the  national  defence  strategy,  force 
structure,  infrastructure,  budget  plan,  and  other  elements  of  the  defence  program  and  policies  of  Norway  with 
a  view  toward  determining  and  expressing  a  defence  strategy  and  establishing  a  defence  program  for  the  next 
20  years.  SDRs  have  been  conducted  on  several  occasions,  the  last  three  in  2000,  2003,  and  2007.  Norway  will 
continue  using  SDRs  in  the  future,  but  will  change  to  a  more  continuous  long-term  defense  planning  process. 
The  scope  of  most  of  the  studies  will  be  narrower  and  conducted  continuously,  with  a  large  review  every  forth 
year  to  sum  up  the  development.  Based  on  the  SDR/  decisions  in  the  Parliament,  the  NMoD  develops  a  Long 
Term  Development  Plan  (LTDP)  where  the  main  goal  is  to  provide  strategic  guidance  and  to  ensure  and  maintain 
a  close  link  between  long  term  planning  and  the  ongoing  implementation.  Another  purpose  of  the  LTDP  is  to 
serve  as  a  coherent  long-term  plan  for  organization,  personnel,  material  and  infrastructure.  It  gives  the  NMoD 
and  the  Armed  Forces  multi  annual  planning  guidelines,  which  again  provide  a  basis  for  annual  planning.  The 
LTDP  has  four  important  annexes: 

•  Structural  Development  Plan 
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•  Long-term  Material  Plan 

•  Long-term  Infrastructure  Plan 

•  Long-term  Personnel  Plan 


Overarching  framework  for  defence  planning 
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Figure  10:  Norway’s  overall  picture  of  the  defence  planning. 

The  model  depicted  in  Figure  10  shows  the  overall  picture  of  the  defence  planning  and  how  factors  from 
areas  such  as  strategic  analysis,  defence  policy  targets  and  priorities  give  inputs  and  guidance  to  the  planning 
process.  Although  it  shows  a  clear  distinction  between  the  different  areas,  the  reality  is  different.  It  is  usually 
not  easy  to  say  where  the  borderline  is  between  e.g.  strategic  analysis  and  the  planning  process.  Furthermore 
the  processes  outside  the  “black  box”  are  just  as  dependent  of  the  planning  process  as  vice  versa. 

Sweden 

Approximately  every  third  year  the  Government  give  directions  regarding  the  Armed  Forces  in  a  Defence  Policy 
Bill  followed  by  a  Parliamentary  Decision.  The  Defence  Policy  Bill  covers  strategic  analysis  about  development 
in  the  geographical  vicinity,  areas  of  interest  for  future  operations  and  guidelines  for  the  composition  of  the  units 
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in  the  Armed  Forces.  The  impact  is  mainly  on  the  midterm  defense  planning.  Capabilities  are  not  used  as  a  part 
of  the  strategic  framework.  With  the  same  frequency  as  the  Defence  Policy  Bill  the  Armed  Forces  presents  the 
results  of  the  long  term  planning. 


Germany 

In  accordance  with  legal  provisions,  the  equipment  of  the  Bundeswehr  must  be  geared  towards  its  mission 
as  defined  by  political  decisions  and  the  tasks  derived  there  from.  The  economic  use  of  resources  demands 
the  exclusive  orientation  to  this  mission.  In  order  to  be  able  to  accomplish  its  mission,  the  Bundeswehr  must 
have  certain  capabilities.  Potential  capability  gaps — existing  or  arising — are  identified  by  means  of  a  capability 
analysis  across  all  organizations  and  task  areas.  To  close  these  capability  gaps,  potential  solutions  are  to  be 
investigated  within  all  planning  categories  (organization,  personnel,  armaments,  maintenance  and  operation,  as 
well  as  infrastructure).  Closing  a  capability  gap  in  the  planning  category  "armaments"  (materiel  solution)  re¬ 
quires  activities  aimed  at  determining  and  meeting  the  demand.  The  implementation  of  these  measures  must  not 
exceed  the  scope  of  resources  available  within  the  foreseeable  future.  Due  to  its  high  innovative  speed,  private 
industry  sets  the  pace  in  technological  development.  Therefore,  close  cooperation  between  the  Bundeswehr  and 
industry  is  absolutely  necessary  in  order  to  be  able  to  maintain  modem  and  efficient  armed  forces. 


Turkiye 

Goal  settings  and  global  analysis  can  be  found  in  Turkish  National  Military  Strategy  which  is  signed  by  Pres¬ 
ident  of  Republic,  Prime  Minister,  Ministry  of  National  Defense  and  General  Stuff  of  Turkish  Armed  Forces 
(TAF).  National  Defense  Analysis  is  described  in  TAF  Operational  Requirements  Plan  which  is  written  by  mili¬ 
tary  only.  This  document  is  updated  in  every  even  year.  The  Strategic  Target  Plan  includes  the  trends  in  national 
defense  which  sometimes  has  application  of  CPA  according  to  technological  development  in  the  country  and  is 
updated  in  every  odd  year.  Strategy  formulation  is  defined  in  The  Strategic  Target  Plan  in  the  next  decade  with 
an  update  in  2  years  time.  If  it  includes  scenarios  which  are  also  tested  by  MODSIM,  10  Year’s  Acquisition 
Plan  document  is  also  updated  according  to  the  results. 

5.4  Needs  and  Solutions 

Areas  of  analysis 

•  Needs  and  solutions 

•  Use  of  capability  based  planning 

•  Overview  of  the  process 

•  Years  and  frequency  of  coverage 

•  Use  of  scenarios 

•  Responsibility 
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Figure  11:  Summary  of  analysis  of  nations’  responses  with  respect  to  defence  needs  and  solutions. 


Summary 

Most  of  the  sample  nations  describe  their  process  to  identify  needs.  They  all  use  capabilities  as  a  part  of  defense 
planning  and  execute  a  gap  analysis.  All  nations  seem  to  have  alternative  solutions  that  in  different  ways  are 
prioritized  before  any  decisions  for  future  acquisition  are  made.  Scenarios  are  for  identifying  needs  or  for 
describing  strategic  frameworks  or  both.  This  also  means  that  some  scenarios  are  not  as  detailed  as  others.  The 
responsibility  for  working  with  needs  and  solutions  differs  between  the  nations.  This  is  due  to  the  different 
governmental  organizations  in  each  country. 

United  States 

The  U.S.  employs  a  capabilities-based  framework  called  the  Joint  Capabilities  Integration  and  Development 
System  (JCIDS)  for  identifying  and  analyzing  needs  and  the  Defense  Acquisition  System  (DAS)  for  developing 
and  acquiring  materiel  solutions,  with  the  connection  between  the  two  processes  shown  in  Figure  12. 

Identification  of  Military  Needs 

JCIDS  is  the  process  by  which  DoD  identifies,  assesses,  and  prioritizes  what  capabilities  the  military  requires  to 
fulfill  its  mission.  JCIDS  is  often  referred  to  as  the  requirements  generation  process.  The  requirements  identified 
through  JCIDS  can  be  addressed  in  a  number  of  ways,  including  changes  in  doctrine,  training,  organization,  or 
the  acquisition  of  a  new  weapon  system. 

JCIDS  was  created  in  2003  in  an  effort  to  fundamentally  change  the  way  the  Department  develops  re¬ 
quirements.  Before  JCIDS,  DoD  used  a  threat-based  approach  to  identify  warfighter  requirements  based  on 
who  the  adversary  might  be  or  where  a  war  might  be  fought.  With  the  advent  of  JCIDS,  DoD  shifted  to  a 
capabilities-based  approach  to  identifying  warfighter  needs  based  on  how  an  adversary  might  fight. 

JCIDS  is  overseen  by  the  Joint  Requirements  Oversight  Council  (JROC)  and  supports  the  Chairman  of  the 
Joint  Chiefs  of  Staff,  who  is  responsible  for  advising  the  Secretary  of  Defense  on  the  priorities  of  military  re¬ 
quirements  in  supporting  the  national  military  strategy.  Within  JCIDS,  Functional  Capability  Boards  (FCBs) 
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Continuous  Technology  Development  and  Maturation 


Figure  12:  U.S.  capabilities-based  framework. 

manage  different  capability  area  portfolios. 1 1  They  are  headed  by  a  general  or  an  admiral  and  made  up  of  mili¬ 
tary  and  civilian  representatives  from  the  military  services,  Joint  Staff,  combatant  commands,  and  the  Office  of 
the  Secretary  of  Defense.  The  FCBs  are  intended  to  support  the  JROC  by  evaluating  capability  needs,  recom¬ 
mending  enhancements  to  capabilities  integration,  examining  joint  priorities,  assessing  program  alternatives, 
and  minimizing  duplication  of  effort  across  the  Department. 

The  JCIDS  process  requires  the  identification  of  gaps  in  military  capabilities.  Potential  solutions  for  filling 
these  gaps  are  identified  and  analyzed  within  DOTMLPF  solution  space.12  The  results  of  these  analyses  or 
capability-based  assessments  (CBAs)  are  formally  submitted  as  initial  capabilities  documents,  or  ICDs.  An 
ICD  is  a  capability  proposal  by  a  military  service,  defense  agency,  combatant  command,  FCB,  or  other  sponsor. 
ICDs  are  intended  to  document  a  specific  capability  gap  or  set  of  gaps  that  exist  in  joint  warfighting  functions 
and  propose  a  prioritized  list  of  various  solutions  to  address  the  gaps.  When  a  capability  proposal  is  submitted, 
a  Joint  Staff  “gatekeeper”  conducts  an  initial  review  to  determine  what  level  of  joint  interest  and  review  there 
should  be  and  which  FCB  should  take  the  lead.  Capability  proposals  deemed  to  have  a  significant  impact 
on  joint  warfighting,  such  as  those  involving  potential  major  defense  acquisition  programs,  are  designated  as 
“JROC  interest”  and  must  be  validated  or  approved  by  the  JROC. 

The  JROC  may  approve  an  ICD  and  recommend  a  non-materiel  solution,  such  as  a  change  to  strategy  or 
tactics.  If  the  JROC  approves  the  pursuit  of  a  materiel  solution1 3  the  program  enters  the  Defense  Acquisition 
System.  Documentation  developed  during  the  JCIDS  process  is  used  throughout  the  acquisition  process. 

1  'The  JROC  charters  eight  FCBs  to  perform  analyses:  Command  and  Control;  Battlespace  Awareness;  Force  Application;  Logistics; 
Protection;  Force  Support;  Net  Centric;  and  Building  Partnerships. 

12Doctrine,  organization,  training,  materiel,  leadership  and  education,  personnel  and  facilities. 

I3A  materiel  solution  is  any  item  (including  ships,  tanks,  self-propelled  weapons,  aircraft,  etc.,  and  related  spares,  repair  parts,  and 
support  equipment,  but  excluding  real  property,  installations,  and  utilities)  necessary  to  equip,  operate,  maintain,  and  support  military 
activities  without  distinction  as  to  its  application  for  administrative  or  combat  purposes. 
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ORGANIZATION 


Development  and  Procurement  of  Materiel  Solution 

DoD  purchases  goods  and  services  from  contractors  to  develop  and  maintain  the  force,  including  related  in¬ 
frastructure.  Any  purchase  of  a  good  or  service  by  DoD  is  defined  as  a  procurement.  In  contrast,  the  term 
defense  acquisition  is  a  broader  term  that  applies  to  more  than  just  the  purchase,  or  procurement,  of  an  item 
or  service.  The  acquisition  process  encompasses  the  design,  engineering,  construction,  testing,  deployment, 
sustainment,  and  disposal  of  weapons  or  related  items.  More  formally,  the  Defense  Acquisition  System  for 
designing  and  acquiring  materiel  solutions  is 


"...  the  management  process  by  which  the  Department  of  Defense  provides  effective,  afford¬ 
able,  and  timely  systems  to  the  users,  [and  it]  exists  to  manage  the  nationOs  investments  in  tech¬ 
nologies,  programs,  and  product  support  necessary  to  achieve  the  National  Security  Strategy  and 
support  the  United  States  Armed  Forces.” 


As  depicted  in  Figure  13,  the  Defense  Acquisition  System  uses  “milestones”  to  oversee  and  mange  acquisi¬ 
tion  programs.  Each  milestone  has  specific  requirements.  A  program  must  meet  these  requirements  (statutory 
and  regulatory)  in  order  to  proceed  to  the  next  phase  in  the  acquisition  process.  The  Milestone  Decision  Au¬ 
thority  (MDA)  is  responsible  for  deciding  whether  a  program  meets  the  milestone  criteria.  Depending  on  the 
program,  the  MDA  can  be  the  Office  of  the  Undersecretary  of  Defense  (Acquisition,  Technology,  &  Logistics) 
or  an  acquisition  executive  in  a  DoD  component,  such  as  a  Military  Department. 


•  The  Materiel  Development  Decision  precedes  entry  into  any 
phase  of  the  acquisition  management  system 
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Figure  13:  U.S.  defense  acquisition  system  milestones. 
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To  enter  the  Defense  Acquisition  System,  a  program  must  pass  a  Materiel  Development  Decision  (MDD) 
review,  chaired  by  a  Milestone  Decision  Authority.  At  the  review,  JROC  recommendations  are  presented  by  the 
Joint  Staff,  and  the  relevant  component  presents  the  Initial  Capabilities  Document,  which  details  the  operational 
need  for  a  materiel  solution.  During  the  Materiel  Solution  Analysis  phase,  an  Analysis  of  Alternatives  (AoA) 
is  conducted  and  a  Technology  Development  Strategy  (TDS)  is  created.  The  Technology  and  Development 
phase  is  where  a  program  determines  what  technologies  are  required  to  develop  a  materiel  solution,  and  works 
to  mature  those  technologies.  This  phase  is  also  where  competitive  prototyping  occurs.  The  Engineering 
and  Manufacturing  Development  phase  is  where  a  system  (or  increment)  is  developed,  full  system  integration 
occurs,  and  preparations  are  made  for  manufacturing,  including  developing  manufacturing  processes,  designing 
for  producibility,  and  managing  cost.  The  Production  and  Deployment  phase  is  where  a  system  is  produced  and 
deployed.  At  Milestone  C,  the  MDA  authorizes  the  beginning  of  low-rate  initial  production  (LRIP),  which 
is  intended  to  both  prepare  manufacturing  and  quality  control  processes  for  a  higher  rate  of  production  and 
provide  production  representative  articles  for  operational  test  and  evaluation  (OT&E).  Upon  completion  of 
OT&E  and  demonstration  of  adequate  control  over  manufacturing  processes,  and  with  the  approval  of  the  MDA, 
a  program  can  go  into  full  rate  production.  When  enough  systems  arc  delivered  and  other  pre-defined  criteria 
are  met,  an  Initial  Operating  Capability  (IOC)  can  be  attained,  allowing  for  some  degree  of  operations.  Full 
Operational  Capability  (FOC)  is  achieved  when  the  system  is  ready  to  operate  as  required.  After  development 
and  production,  a  system  enters  the  sustainment  phase.  Finally,  programs  are  divided  into  acquisition  categories 
(ACATs)  based  primarily  on  program  value.  Management  and  oversight  of  acquisition  programs  increases  as 
the  value  of  the  program  increases.  The  most  significant  DoD  and  Congressional  oversight  activities  apply  to 
major  defense  acquisition  programs  (MDAPs),  which  are  categorized  as  ACAT  Is. 


Canada 

Canada,  in  their  template,  has  described  a  process  focusing  on  needs,  not  how  to  develop  solutions.  The 
process  is  capability  based.  The  strategic  decision-making  process  for  defense  acquisitions  is  called  the  Force 
Development  (FD)  process  and  the  output  is  the  Investment  Plan.  The  Investment  Plan  covers  15  years  with  the 
first  5  years  in  detail.  The  plan  provides  a  rationale  for  new  projects  and  funding  provision  of  them. 


Figure  14:  SAS-076’s  interpretation  of  Canada’s  force  development  process. 


The  process  starts  with  development  and  analysis  of  scenarios  that  are  representative  for  future  missions. 
Starting  with  the  scenarios  it  is  possible  to  determine  the  relative  importance  of  different  capabilities  and  priori¬ 
tize  the  capabilities  for  investment,  sustainment  or  divestment.  Courses  of  action  for  priority  areas  and  selected 
preferred  options  are  developed  and  proposed  program  changes  are  integrated  with  existing  programs.  Finally 
an  integrated  Departmental  resource  plan  (Investment  Plan)  is  developed.  Multiple  planning  scenarios  are  used 
to  determine  likelihood  of  use  and  criticality  of  delivery  of  different  capabilities.  Environmental  Chiefs  of  Staff, 
Chief  of  Force  Development  and  Chief  of  Program  work  within  the  process. 
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The  Netherlands 

The  Netherlands,  in  their  template,  has  described  a  process  focusing  on  solutions,  not  how  to  develop  needs. 
Capabilities  seem  to  be  a  part  of  the  process  although  it  is  not  described  in  what  way  they  are  used.  The  process 
used  to  identify  and  acquire  the  different  solutions,  i.e.  different  materiel,  is  called  Defence  Material  Process 
(DMP).  The  DMP  is  divided  into  several  phases:  each  phase  is  ended  with  a  DMP-document. 


Figure  15:  SAS-076’s  interpretation  of  Netherlands’  defence  material  process. 

DMP  is  the  guideline  for  keeping  the  materiel  management  process  on  track  and  contains  rules  for  realizing 
the  requirement  for  materiel  procurement.  Financial  claims  are  described.  The  preliminary  study  results  in  an 
initial  selection  of  product  alternatives,  including  a  short  list  of  most  suitable  alternatives.  The  Study  phase 
should  result  in  a  validation  or  reassessment  of  qualitative  &  quantitative  requirements,  the  detailed  require¬ 
ments  and  norms  the  product  must  comply  with  and  used  to  compare  and  evaluate  alternatives;  an  estimate  of 
the  life-cycle  costs  of  each  alternative  product  and  the  financial  implications  of  the  alternatives  and,  if  necessary 
in  a  development  process  or  a  prototype  trial,  the  outcome  of  the  tenders.  Preparation  for  procurement  results 
in  a  choice  of  a  particular  product  and  the  work  includes  validation  or  reassessment  of  requirements,  results  of 
development,  specification  of  estimated  project  expenditure  resulting,  etc.  The  final  phase,  Evaluation,  applies 
for  projects  exceeding  Euro  250  million  and  is  applied  when  a  system  has  been  in  use  for  some  time.  The  par¬ 
liament  is  informed  of  the  evaluation.  The  DMP  provides  for  political  decision  making  at  important  milestones 
and  political  and  non-political  steering  throughout  duration  of  project. 


Norway 

Norway,  in  their  template,  has  described  a  process  focusing  on  needs,  not  how  to  develop  solutions.  The 
process  to  identify  military  needs  is  capability  based  and  called  Capability  Planning.  The  process  maintains  a 
link  between  defense-  and  security  policy  foundation,  defense  tasks  and  level  of  ambition,  and  main  decisions 
concerning  development  of  capabilities  and  how  Defense  Forces  are  structured.  The  Capability  Planning  starts 
with  Task  analysis  related  to  tasks  stated  in  parliamentary  bill.  A  complete  list  of  capabilities  needed  to  solve 
imposed  tasks  is  developed  and  then  the  level  of  ambition  is  stated.  The  Gap  analysis  identities  and  analyzes 
the  gap  between  the  ability  the  structure  has  today  and  the  defined  capability  need.  A  prioritizing  is  made  based 
on  probability,  impact  and  economy.  Finally  the  capability  development  plan  is  developed.  The  plan  describes 
the  gaps  that  are  prioritized  and  in  what  way  they  will  be  closed. 
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Figure  16:  SAS-076’s  interpretation  of  Norway’s  capability  planning  process. 
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The  Capability  Development  Plan  is  used  to  adjust  the  Structural  Development  Plan  which  is  part  of  the 
Long  Term  Development  Plan  (LTDP).  LTDP  is  a  result  of  Strategic  Defense  Reviews  which  historically  have 
been  made  every  third  or  fourth  year.  Besides  Capability  Planning  there  is  Structural  Planning  to  maintain 
development  of  Defense  Forces  within  defined  lines  of  development  and  capability  requirements  and  acquire 
capabilities  through  decisions  on  structural  development  and  planned  allocation  of  resources.  Use  of  scenarios 
is  not  specified  although  they  are  used  and  their  role  in  defense  planning  has  been  described  earlier  (example: 
conference  in  Paris).  NMoD  is  responsible  for  the  work  with  support  from  the  Norwegian  Defense  Research 
Establishment  (NDRE). 

Sweden 

Sweden,  in  their  template,  has  described  a  process  focusing  on  needs,  not  how  to  develop  solutions.  It  is  not 
explicit  in  the  answer  but  capabilities  are  a  part  in  the  defense  planning.  But  capabilities  are  not  the  only  way 
to  identify  needs  and  a  more  capability  based  approach  is  discussed. 


Figure  17:  SAS-076’s  interpretation  of  Sweden’s  defence  planning  process. 

The  defence  planning  process  starts  with  an  analysis  in  order  to  describe  the  difference  between  the  current 
plan  and  future  needs.  The  difference  between  the  plan  and  the  identified  future  needs  is  the  basis  for  the 
changes  that  needs  to  be  done  within  the  military  units.  The  units  are  described  considering  for  example 
number,  quality  and  costs  related  to  personnel,  acquisition  and  exercises  etc.  All  changes  are  prioritised.  The 
development  plan  and  the  budget  proposal  must  be  funded,  this  is  done  by  balancing  the  plan.  When  the  plan 
is  balanced  it  is  possible  to  put  together  the  development  plan  and  the  budget  proposal.  The  development  plan 
contains  information  about  the  number  of  military  units,  acquisition  of  new  systems,  a  plan  for  ongoing  and 
coming  operations,  and  so  on.  The  acquisition  planning  cycle  starts  from  decisions  taken  in  the  development 
plan.  The  midterm  defence  planning  covers  the  time  frame  up  to  ten  years  and  it  is  normally  revised  once 
a  year.  The  result  of  the  defence  planning  is  the  Armed  Forces’  development  plan  and  a  budget  proposal. 
The  development  plan  describes  in  detail  the  Armed  Forces  for  the  first  three  years  in  the  period  and  work 
as  a  guideline  for  the  remaining  years.  Scenarios  are  used  to  prioritize  capabilities.  The  Armed  Forces  are 
responsible  for  the  process. 


Germany 

The  information  regarding  the  German  process  describes  both  needs  and  solutions  but  is  focused  on  the  so¬ 
lutions.  The  process  is  capability  based.  Potential  capability  gaps  are  identified  by  capability  analysis  across 
organizations  and  tasks.  Potential  solutions  to  close  capability  gaps  can  be  identified  not  only  by  acquisition  of 
new  material  but  with  a  changed  organization,  infrastructure  etc.  Implementation  of  solutions  must  not  exceed 
the  scope  of  resources  available.  Customer  Product  Management  (CPM)  is  the  process  used  to  determination 
and  meeting  the  capabilities  needed.  The  process  is  determined  by  principles  of  cost-effectiveness. 
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ORGANIZATION 


Figure  18:  SAS-076’s  interpretation  of  Germany’s  customer  product  management  process. 

Determination  and  meeting  of  demands  is  divided  into  the  phases  analysis,  risk  reduction,  introduction  and 
in-service  phase.  During  the  analysis  phase,  the  demand  is  determined  and  information  to  be  able  to  select  and 
implement  the  selected  solution  is  gathered.  Risk  reduction  means  that  all  necessary  steps  is  taken  to  ensure  that 
risks  relating  to  performance,  time  and  cost  aspects  are  reduced.  During  the  introduction  phase,  development 
work  will  be  rendered  as  part  of  a  purchase — or  manufacture-  (construction-) — based  contract  and  serves  the 
purpose  of  adapting  the  solution  to  the  requirements  and  preparing  it  for  production.  Development  activities 
during  the  in-service  phase  serve  to  maintain  initial  operational  capability.  The  Bundeswehr  Chief  of  Staff 
determines  required  Bundeswehr  capabilities,  the  Director  General  of  Armaments  and  the  IT  Director  are  in¬ 
volved  in  the  performance  of  the  technical  and  economic  assessment.  The  Chiefs  of  Staff  of  the  armed  services, 
the  MOD  Directors  and  the  IT  Director  participate  in  the  capability  analysis.  The  Director  General  of  Arma¬ 
ments,  the  IT  Director  and  the  Director  of  Defense  Administration  each  assume  ministerial  responsibility  for 
the  provision  of  the  appropriate  products  and  services  required  to  close  capability  gaps  within  their  respective 
areas  of  responsibility. 

Turkiye 

“Scientific  Decision  Support”  methods  are  used  in  the  planning  activities  process  in  the  fields  of  operations, 
force,  personnel,  logistics,  intelligence  and  health  as  well  as  in  the  planning  and  programming  of  the  capabilities 
and  systems  that  the  Armed  Forces  must  have  to  ensure  the  defense  and  survivability  of  our  country  according 
to  the  current  and  future  requirements.  The  process  regarding  the  above-mentioned  scientific  decision  support 
methods  is  basically  composed  of  the  following  stages: 

•  Definition  of  the  need  for  analysis/question, 

•  Collection  and  compilation  of  the  required  data, 

•  Making  analyses  through  analytical  models  and  simulation  systems, 

•  Generation  of  decision  alternatives  and  their  presentation  to  the  decision-making  authority. 

The  outputs  of  needs  &  solutions  analysis  which  is  done  by  “Scientific  Decision  Support”  are  mentioned  in 
“Strategic  Target  Plan”  as  the  needs  required  and  “10  Years’  Acquisition  Plan”  as  the  solutions.  Both  documents 
cover  next  ten  years  with  updates  in  each  2-year  period  and  the  responsibility  of  these  documents  is  given  to 
General  Stuff  of  TAF. 

5.5  Lexicon  and  Taxonomy 

Most  nations  from  the  sample  seem  to  define  capability  in  terms  of  ability  to  achieve  a  desired  outcome.  Re¬ 
garding  components  of  the  capabilities  there  seems  to  be  a  significant  commonality.  The  result  of  the  template 
shows  that  some  nations  use  a  much  more  extensive  structure  than  others. 
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•  Definition  of  a  defense  capability:  Most  nations  seem  to  define  capability  in  terms  of  ability  to  achieve  a 
desired  outcome. 


-  Turkiye: 

Activities  process  in  the  fields  of  operations,  force,  personnel,  logistics, 
intelligence  and  health  as  well  as  in  the  planning  and  programming  of  the 
capabilities  and  systems  that  the  Armed  Forces  must  have  to  ensure  the 
defense  and  survivability  of  our  country  according  to  the  current  and  future 
requirements. 

-  United  Kingdom: 

The  enduring  ability  to  generate  a  desired  operational  outcome  or  effect, 
and  is  relative  to  the  threat,  physical  environment,  and  contributions  of 
coalition  partners. 

-  United  States: 

The  ability  to  achieve  a  desired  effect  under  specified  standards  and  con¬ 
ditions  through  combinations  of  means  and  ways  to  execute  a  specified 
course  of  action. 

-  Canada: 

Identical  to  definition  used  by  the  U.S. 

-  Norway: 

The  ability  to  complete  a  given  task. 

-  Sweden: 

Activities  for  which  units  have  been  acquisitioned  and  trained  to  reach  a 
specific  effect  depending  on  scenario  and  ambition. 

-  The  Netherlands: 

The  only  definition  used  by  The  Netherlands  relates  to  capability,  and 
is  taken  from  the  NATO  Bi-SC  (Strategic  Command)  Agreed  Capability 
Codes  and  Capability  Statements  dated  16  April  2008.  It  reflects  opera¬ 
tional  capabilities  which  the  armed  forces,  or  a  part  thereof,  should  or  can 
have. 

•  Components  of  capability:  There  seems  to  be  significant  commonality  as  shown  here: 


-  Turkiye: 

Doctrine,  Training,  Education,  Leadership,  Organization,  Materiel,  Per¬ 
sonnel 

-  Canada: 

Personnel,  Research  and  Development,  Infrastructure,  Concepts  and  Doc¬ 
trine,  Information,  and  Equipment  (PRICIE) 

-  United  States: 

Doctrine,  Organization,  Training,  Materiel,  Leadership  and  Education, 
Personnel,  and  Facilities  (DOTMLPF) 

-  Norway: 

Protection,  Means  of  Action,  ISTAR,  C2,  Mobility,  Support 

-  Sweden: 

There  is  no  stipulated  list  of  components  of  the  capabilities.  However  they 
can  be  described  in  terms  of  Doctrine,  Organization,  Training,  Materiel, 
Leadership  and  Education,  Personnel,  and  Facilities  or  other  similar  ex¬ 
pressions. 

-  The  Netherlands: 

DCTOMP(F):  Doctrine,  Command,  Training  &  Education,  Organisation, 
Materiel,  Personnel  and  Finance 

•  Taxonomy:  Some  nations  use  a  much  more  extensive  structure  than  others. 
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-  United  States:  Like  capabilities  are  functionally  grouped  into  Joint  Capability  Areas 

(JCAs)  which  cover  100%  of  DoD’s  budget  and  warfighting  capability. 
Tier  1  JCAs  are  broken  down  into  increasingly  finer  degrees  of  detail. 
Eventually,  specific  capabilities  are  linked  to  discrete  warfighting  tasks, 
such  as  neutralizing  enemy  sea  mines  in  Very  Shallow  Water.  Capabilities 
and  associated  tasks  number  in  the  thousands. 

-  Canada:  The  Canadian  Capability  Framework  comprises  five  capability  domains: 

Act,  Shield,  Sustain,  Command  and  Sense.  These  are  each  expressed  as 
a  hierarchical  tree  of  capabilities  that  cover  all  of  the  operational  outputs 
of  the  Canadian  Forces.  The  five  domains  break  down  to  336  discrete 
capabilities.  A  sixth  domain,  Generate,  is  identified  as  covering  all  of  the 
non-operational  capabilities  of  the  Department  of  National  Defense  and  the 
Canadian  Forces  (DND/CF),  but  has  not  yet  been  further  developed. 

-  Norway:  Broad  capabilities  are  defined  for  each  of  the  six  major  domains.  Exam¬ 

ples  are  Protection-Land-Heavy  Machine  Gun  and  Support- Joint-Medical 
Services. 

-  Sweden:  The  Swedish  Armed  Forces  uses  a  model  with  Core  Tasks,  Defence  Tasks 

and  Defence  Capabilities.  The  different  tasks  and  capabilities  are  used  to 
help  the  organisation  to  describe  mission  of  the  Armed  Forces.  An  example 
of  a  Core  Task  is  protecting  the  integrity  of  national  territory,  an  example 
of  a  Defence  Task  is  guarding  maritime  territory  and  finally  an  example  of 
a  Defence  Capability  is  effect  on  sea  targets.  There  arc  approximately  80 
capabilities. 

5.6  Role  of  Life-Cycle  Cost  Analysis  in  Defence  Planning 
Areas  of  analysis 

The  responses  of  each  of  the  nations  were  analyzed  with  respect  to  the  role  of  life-cycle  cost  analysis  during 
three  phases  of  defence  planning  and  acquisition: 

•  Identification  of  needs; 

•  Programming  and  budgeting;  and, 

•  Acquisition  of  materiel  solutions. 

For  each  of  these  phases,  SAS-076  analyzed  the  nation’s  response  in  terms  of: 

•  Requirement  for  LCC  analysis; 

•  Performance  of  LCC  analysis;  and, 

•  Guidelines  for  LCC  analysis. 

Furthermore,  when  it  was  determined  that  LCC  analysis  is  performed,  SAS-076  analyzed  whether  the  results 
of  the  LCC  analysis  were  used  by  the  nation  for: 

•  Economic  information; 
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•  Prioritization; 

•  Affordability  assessment;  or, 

•  Analysis  of  alternatives. 

Summary 

Of  the  seven  nations  answering  the  template  only  one  nation  uses  life-cycle  costing  both  for  identifying  defense 
needs,  for  defense  programming  and  budgeting  and  for  acquisition  of  materiel  solutions.  The  best  commonality 
is  identified  in  the  field  of  acquisition  of  materiel  solutions. 

Figures  19  and  20  provide  a  high-level  summary  of  the  SAS-076  analysis.  As  indicated  by  the  legend  in 
Figure  19,  colours  are  used  to  evaluate  the  degree  of  use  and  role  played  played  by  life-cycle  cost  analysis 
during  the  various  stages  of  defence  planning  and  acquisition.  Green  indicates  that  life-cycle  cost  analysis  is 
usually  performed  and  plays  a  role  in  decision  making.  Red  indicates  the  opposite — life-cycle  costing  does  not 
play  a  role.  Yellow  indicates  that  the  use  of  life-cycle  costing  is  occasional  or  the  that  the  nation’s  response  was 
not  concrete.  Question  mark  entries  (“?”)  indicate  that  the  nation’s  response  was  unclear  or  did  not  provide 
enough  information  to  evaluate.  Figure  20  illustrates  the  micro  analysis  which  is  then  combined  to  obtain 
Figure  19. 


Yes 

Occasional  /  Not  Concrete 
No 

Response  unclear 

MACRO  ANALYSIS: 

Identification  of  Defence  Needs 
Defence  Programming  and  Budgeting 
Acquisition  of  Materiel  Solutions 


U.S.  NLD  CAN  SWE  NOR  GER  TUR 


Legend: 


H 


Figure  19:  Macro  analysis  of  role  of  life-cycle  cost  analysis  in  defence  planning. 


United  States 

The  center  of  gravity  of  life-cycle  cost  analysis  in  the  U.S.  is  in  support  of  acquisition  of  materiel  solutions. 
Little  or  no  support  is  provided  for  strategic  planning  and  only  ad  hoc  support  during  the  identification  of 
defence  needs.  Guidance  for  this  latter  phase  merely  recommends  the  execution  of  affordability  assessments. 
Life-cycle  cost  analysis  supports  budgeting  more  than  programming.  Due  to  recent  statutory  requirements, 
the  U.S.  defence  budget  now  needs  to  reflect  the  acquisition  cost  estimates  from  each  of  the  services.  The 
acquisition  of  materiel  solutions  is  heavily  and  extensively  supported  and  informed  by  cost  analyses  which  are 
required  at  milestones  A,B,C,  and  full-rate  production  decision  reviews.  Over  1,000  government,  in-house  cost 
analysts  provide  this  support. 

Norway 

Norway,  one  of  the  very  few  countries  in  the  world  with  a  national  monetary  surplus  or  negative  national 
debt,  sets  the  standard  within  NATO  for  use  of  life-cycle  cost  analysis  early  in  the  strategic  planning  process 
(identification  of  defence  needs).  This  Norwegian  penchant  for  sound  fiscal  constraint  and  avoidance  of  budget 
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MICRO  ANALYSIS: 

Identification  of  Defence  Needs 
Mandatory? 

Performed? 

Follows  Guidelines? 

Used  Economic  info 
Prioritization 
Affordability  assessment 
Analysis  of  Alternatives 


Defence  Programming  and  Budgeting 
Mandatory? 

Performed? 

Follows  Guidelines? 

Used  Economic  info 
Prioritization 
Affordability  assessment 
Analysis  of  Alternatives 


Acquisition  of  Materiel  Solutions 
Mandatory? 

Follows  Guidelines? 
Performed? 

Used  Economic  info 
Prioritization 
Affordability  assessment 
Analysis  of  Alternatives 


Figure  20:  Micro  analysis  of  role  of  life-cycle  cost  analysis  in  defence  planning. 


deficits  helps  ensure  that  national  will  matches  national  wallet.  Norway  believes  that  it’s  all-important  to  get 
the  costs  right  the  first  time  and  simply  not  to  develop,  build,  and  deploy  systems  that  are  unaffordable.  Along 
these  same  lines,  life-cycle  costs  analysis  extensively  supports  the  acquisition  of  materiel  solutions,  either  new 
developments  or  COTS  procurements.  Costs  and  capability  tradeoffs  are  considered,  as  well  as  operating  and 
support  costs  during  the  in-service  phase  of  a  program's  life  cycle. 


Sweden 

In  Sweden,  force-structure  cost  analyses  is  conducted  during  the  long  term  strategic  planning  process.  To 
direct  the  force  structure  planning  process,  the  Swedish  Ministry  of  Defence  can  give  indicative  levels  for  total 
future  expenditures.  The  Swedish  defence  planning  process  analyzes  the  difference  between  the  current  and 
future  needs.  The  difference  between  the  two  provides  the  basis  for  changes  and  the  expenditures  that  these 
changes  will  induce  are  estimated.  The  sum  of  all  suggested  changes  are  then  balanced  against  each  other  in 
the  exercise  called  balancing  the  plan  resulting  in  a  new  development  plan.  All  changes  are  prioritised  so  it 
is  possible  to  choose  which  changes  to  start  with  if  there  is  not  enough  budget  to  do  them  all.  There  is  an 
equipment  acquisition  strategy  which  stipulates  that  the  equipment  acquisition  should  be  cost  efficient  from  a 
LCC-perspective,  and  there  is  a  directive  stipulating  how  the  LCC-estimates  should  be  prepared. 

The  greatest  percentage  of  resources  in  the  cost  community  is  devoted  to  support  the  defense  acquisition 
system,  but  there  arc  no  mandatory  directives  in  the  field  of  cost  estimation  and  analysis,  only  advisory  and 
instructional  policies.  Further  cost  analysis  support  is  provided  for  development  and  optimization  of  support 
systems,  of  the  systems  maintainability  and  reliability,  of  contractor  source  selection  and  earned  value  manage¬ 
ment. 
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The  Netherlands 

Netherlands  is  perhaps  the  only  country  in  NATO  that  takes  a  “whole  of  government”  (WoG)  approach  to 
national  security  planning.  Strategic  requirements  and  identification  of  needs  and  solutions  cover  the  entire 
spectrum  of  national  defence  and  civil  concerns.  Like  the  U.S.,  defence  cost  analysis  supports  acquisition  of 
materiel  solutions  very  extensively.  This  often  involves  make-or-buy  decisions  for  weapon  systems  based  on 
their  estimated  life-cycle  costs.  Furthermore  cost  and  capability  tradeoffs  are  commonplace.  In  the  earlier 
stages  of  national  planning,  cost  analysis  employed  far  less  frequently.  For  example,  elements  of  risk  (e.g., 
breach  of  dyke  system  or  occurrence  of  a  pandemic  virus)  are  analyzed  in  terms  of  probability  of  occurrence 
and  consequence.  Cost,  however,  is  not  a  dimension  of  the  analysis  at  this  early  stage  of  planning. 

Canada 

As  part  of  Canada’s  capability-based  planning  process,  life-cycle  costs  are  taken  into  consideration  during  the 
identification  of  defence  needs  phase  of  defence  planning.  At  this  early  stage  cost  estimates  rough  order  of 
magnitude  calculations  based  on  PRICIE  cost  breakdown  models  are  employed.  During  the  programming  and 
budgeting  phase — Canada’s  integration  phase  of  its  capability-base  planning  process — these  rough  order  mag¬ 
nitude  costs  are  considered  during  the  decision  process  and  used  for  prioritization  of  alternatives  and  further 
economic  planning.  Prioritization  and  analysis  of  alternatives  is  subjective.  As  with  other  NATO  countries,  the 
majority  of  Canada’s  life-cycle  cost  analysis  is  focused  during  the  acquisition  stages.  The  Canadian  Department 
of  National  Defence  must  submit  detailed  cost  estimates  to  the  Treasury  Board  at  various  program  stages  (e.g., 
preliminary  and  effective  project  approval  stages)  in  order  to  obtain  funding  approval.  Comparative  life-cycle 
cost  models  are  employed  by  certain  acquisition  programs  in  order  to  evaluate  industry  proposals,  however  life- 
cycle  costs  are  often  appropriated  less  importance  than  the  acquisition  cost  during  tender  selection.  Canada’s 
Department  of  National  Defence  does  not  have  a  dedicated  cost  analysis  group.  Acquisition  program  offices 
typically  employ  a  small  team  of  financial  analysts  who  generate  life-cycle  cost  estimates.  Occasionally  ex¬ 
ternal  consulting  companies  are  contracted  to  generate  primary  and/or  secondary  acquisition  cost  estimates. 
Final  program  estimates  are  checked  and  validated  internally,  although  the  level  of  scrutiny  or  application  of 
guidelines  is  unclear. 

Turkiye 

Turkiye,  one  of  the  countries  within  NATO  which  has  a  biggest  defence  budget  according  to  the  percentage  of 
GDP  (5-6%),  surprisingly  has  no  specific  life-cycle  cost  analysis  in  the  strategic  planning  process  for  acquiring 
the  defence  capability.  The  only  use  of  the  life-cycle  costing  in  macro  analysis  is  during  defence  programming 
and  budget  which  is  done  occasionally.  During  identification  of  defence  needs,  the  life-cycle  costing  is  used 
only  for  economic  input  to  make  prioritization  and  sometimes  for  analysis  of  alternatives.  In  Turkiye,  the 
mostly  occasional  use  of  life-cycle  costing  analysis  is  in  the  acquisition  phase  for  economical  prioritization  of 
procurement  alternatives.  Despite  the  use  of  life-cycle  costing  analysis  in  acquisition,  there  is  no  requirement 
or  obligation  of  industry  to  do  life-cycle  cost  analysis  as  part  of  the  tender  solicitation  from  industry. 
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6.0  BEST  PRACTICES  AND  RECOMMENDATIONS 

Figure  21  indicates  the  likely  current  practice  of  NATO  and  PfP  nations,  with  respect  to  the  role  of  life-cycle 
cost  analysis  in  defence  planning,  as  culled  by  SAS-076  from  national  template  surveys  (based  on  responses  of 
seven  nations). 


Current  Practice  of  NATO  Nations 
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Figure  21 :  Current  practice  of  NATO  nations  with  respect  to  the  role  of  life-cycle  cost  analysis  in  defence  planning. 

Based  on  responses  from  seven  countries,  and  through  a  process  of  vigorous  discussion  and  even  heated 
debate,  SAS-076  identified  the  following  best  practices  with  respect  to  the  role  played  be  cost  analysis  in 

managing  the  defence  enterprise. 
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1.  Engage  Early-On: 

In  the  long  run,  military  power  is  a  function  of  economic  power.  Spending  imprudently  in  the  short  run  on 
unaffordable  “gold-plated”  weapon  systems  will  add  to  budget  deficits  and  national  debt  in  the  long  run. 
Use  cost  assessments  to  inform  the  strategic  decision  planning  process  on  what  a  country  can  and  cannot 
afford.  Canada  and  Scandinavian  countries  seem  the  best  practitioners  on  this  score. 

2.  Budget  and  Program  to  the  Cost  Estimate: 

Independent  cost  estimates  and  cost  assessments  are  valuable  only  to  the  degree  they  are  used.  A  best 
practice  is  to  generate  an  independent  life-cycle  cost  estimate  for  every  major  weapon  system  acquisition 
program  and  to  reflect  this  estimate  in  planning  and  budgeting  systems.  This  is  now  a  mandatory  requirement 
in  the  United  States  Department  of  Defense  for  Acquisition  Category  I  programs. 

3.  Make  Cost  the  Denominator  in  Needs  and  Solutions  Analysis: 

In  identifying  military  needs  and  solutions,  capability-based  planning  is  the  gold  standard  in  NATO  today. 
A  plethora  of  techniques  are  used  to  assess  military  value  of  current  and  prospective  assets  including  known 
performance  characteristics  as  well  as  use  of  opinions  culled  from  subject  matter  experts.  All  too  often 
however,  military  value  is  estimated  and  interpreted  without  a  complete  and  accurate  view  of  life-cycle 
costs.  A  best  practice  is  to  compute  a  return  on  investment  for  defence  assets  defined  as: 

Military  Value 
Life-Cycle  Costs 

It’s  important  to  note  that  risk  and  balance  are  other  essential  dimensions  of  a  complete  analysis. 

4.  Move  to  the  “Left”  in  Acquisition: 

Many,  if  not  all,  countries  extensively  support  the  acquisition  process  at  key  decision  points  (e.g.,  “gates”  in 
the  United  Kingdom,  milestones  in  the  United  States).  However,  many  cost  analysts  object  to  providing  cost 
estimates  early  in  acquisition  when  systems  are  only  vaguely  defined  and  exist  only  in  design.  However, 
paradoxically,  this  is  when  the  cost  estimates  are  of  the  most  importance.  After  all,  it  is  much  easier  to 
cancel  a  bad  program  that  exists  only  in  an  analysis  of  alternatives  than  an  actual  program  struggling  with 
development  or  with  enormous  cost  growth  in  production.  Much  more  extensive  cost-analysis  support  needs 
to  be  provided  in  early  design  phases.  Rough  order  of  magnitude  estimates  a  this  point  in  the  life  cycle  will 
help  empower  informed  decisions  on  the  allocation  of  scarce  defence  assets. 

5.  Optimize  from  an  Alliance  Perspective: 

The  United  States,  Canada,  and  most  European  nations  are  facing  the  dual  dilemmas  of  aging  populations, 
budget  deficits  and  growing  national  debts  that  threaten  stability.  As  a  consequence,  many  nations  are 
cutting  defence  expenditures.  Defence  spending  as  a  percentage  of  Gross  Domestic  Product  is  approaching 
lows  in  contemporary  times  for  many  nations.  Given  pervasive  threat  to  the  security  of  NATO  and  its 
alliance  members  across  the  broad  spectrum  of  conflict,  optimization  of  military  capability  seems  in  order. 
Duplication  of  effort,  such  as  several  countries  possessing  extensive  sea  mine  counter-measure  capability, 
might  be  avoided.  Individual  Alliance  members  might  develop  expertise  in  a  war-fighting  domain  that  could 
be  used  to  the  benefit  of  the  entire  Alliance  in  times  of  conflict.  Savings  of  hundreds  of  billions  of  Euros  per 
annum  might  be  achieved  by  managing  from  a  NATO  enterprise  perspective. 

6.  Present  Decision  Makers  with  Menu  of  Portfolios: 
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Senior  defense  decision  makers  express  a  strong  demand  for  a  menu  of  acquisition  alternatives  from  which 
to  choose.  Heretofore,  inordinate  attention  has  been  devoted,  at  high  levels,  to  the  management  of  individual 
acquisition  programs.  Attention  to  acquisition  details,  at  high  levels,  might  be  better  delegated  to  subordinate 
authorities  in  each  military  department  or  acquisition  community.  Senior  decision  makers,  instead,  would 
focus  on  a  complete  set  or  portfolio  of  systems  in  any  war  fighting  or  mission  area.  The  portfolio  would 
include  science  and  technology  programs,  current  acquisition  programs,  and  operational  systems.  Further, 
alternative  portfolios  could  be  defined.  These  might  include: 

•  A  status  quo  portfolio  of  currently  programmed  systems; 

•  An  international  portfolio  which  would  leverage  assets  from  a  NATO-enterprise  point  of  view;  and 

•  A  high-tech  portfolio  that  would  emphasize  the  development  and  funding  of  science  and  technology 
projects  which  might,  in  the  long  run,  yield  a  competitive  advantage  on  the  battlefield. 

A  review  of  current  and  alternative  portfolios  would  shift  attention  of  senior  decision  makers  to  where  it 
rightfully  belongs,  to  addressing  the  fundamental,  multiple  needs  of  a  ministry  of  defense,  and  trade-off 
space ,  in  the  face  of  an  entire  spectrum  of  threats. 
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